Support Questions
Find answers, ask questions, and share your expertise

How to enable flush per column family instead of Region full flush?

Solved Go to solution

How to enable flush per column family instead of Region full flush?

Super Guru

HBASE-10201 enabled logic to flush based on size of CF. If the size of CF > DEFAULT_HREGION_MEMSTORE_PER_COLUMN_FAMILY_FLUSH then it will flush those CFs and not entire memstore. How to enable this? I don't see much documentation on this feature.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to enable flush per column family instead of Region full flush?

Super Collaborator

There is also hbase.hregion.percolumnfamilyflush.size.lower.bound.min :

If FlushLargeStoresPolicy is used and there are multiple column families, then every time that we hit the total memstore limit, we find out all the column families whose memstores exceed a "lower bound" and only flush them while retaining the others in memory. The "lower bound" will be "hbase.hregion.memstore.flush.size / column_family_number" by default unless value of this property is larger than that. If none of the families have their memstore size more than lower bound, all the memstores will be flushed (just as usual).

Default:

16777216

View solution in original post

3 REPLIES 3

Re: How to enable flush per column family instead of Region full flush?

It is enabled by default(FlushLargeStoresPolicy). You just need to configure below property. It will not flush those column family who size is less than the size specified in below property

hbase.hregion.percolumnfamilyflush.size.lower.bound

Re: How to enable flush per column family instead of Region full flush?

Super Collaborator

There is also hbase.hregion.percolumnfamilyflush.size.lower.bound.min :

If FlushLargeStoresPolicy is used and there are multiple column families, then every time that we hit the total memstore limit, we find out all the column families whose memstores exceed a "lower bound" and only flush them while retaining the others in memory. The "lower bound" will be "hbase.hregion.memstore.flush.size / column_family_number" by default unless value of this property is larger than that. If none of the families have their memstore size more than lower bound, all the memstores will be flushed (just as usual).

Default:

16777216

View solution in original post

Re: How to enable flush per column family instead of Region full flush?

Super Collaborator

Note: hbase.hregion.percolumnfamilyflush.size.lower.bound is used in HDP 2.3 / 2.4

hbase.hregion.percolumnfamilyflush.size.lower.bound.min would be used in HDP 2.5