Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase : Why all the memstore are flushed even one memstore reaches its capacity

Highlighted

HBase : Why all the memstore are flushed even one memstore reaches its capacity

Hi,

Want to understand that, why all the memstore are flushed even one memstore reaches its capacity ?

Is there any property to change this behaviour . If so , what are the pros and cons of it.

Regards, Gobi

1 REPLY 1
Highlighted

Re: HBase : Why all the memstore are flushed even one memstore reaches its capacity

Super Guru

@Gobi Subramani

There are two different properties that you set, that can trigger memstore flush. First is the simple memstore flush size. Memstore flush occurs, when memstore reaches hbase.hregion.memstore.flush.size. In this case only the memstore that reaches this value flushes and not all flushes which is how you rightly think it should work.

However, there is a second reason why a memstore might be flushed. As you might know, a Region server has a number of regions. And each region has a number of stores, each consisting of its own memstore. Now, the memstore, at the end of the day, really lives in Region's heap. Imagine, these memstore, not reaching their memstore flush size, but all together, reaching a certain threshold, which might soon create heap issues for the whole region server. You don't want that situation where the whole region server will be brought down. So, there is a setting. hbase.regionserver.global.memstore.lowerLimit tells region server to flush all memstores if the combined heap usage of all memstores reaches this threshold. In this case, all memstores are flushed.

As number of regions grow and your cluster utilization increases and evenly spread out, you will likely see flushes being done due to this reason, than the first reason.

That's probably the behavior you are witnessing.

Don't have an account?
Coming from Hortonworks? Activate your account here