Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase Compaction Problem

Hbase Compaction Problem

New Contributor

In our Hbase cluster, there is a never ending compaction queue, which is also getting larger day by day. We've changed and edited a lot of parameters, but nothing helped. This situtation is also affecting cluster performance and stability. For this specific problem, we performed a thorough reading on Hbase documentation, and did a lot of tests according to our inquiries, but did not get a satisfactory result.

We are using Hbase as a graph store. On a daily basis, it grows by 1,5 TB/1,5 billion rows. The rows are relatively small size. And the operations are write-intensive.
Our 17 region servers have 256 GB of memory installed on them and we have not observed any bottleneck on servers' resources.

We are using multiple tables, but here are the definitions of tables, which are doing the most of the I/O.

1-
'myGraph:edgeIndices', {TABLE_ATTRIBUTES => {DURABILITY => 'USE_DEFAULT'}, {NAME => 'f', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'SNAPPY', BLOCKCACHE => 'true', BLOCKSIZE => '131072'}

2-
'myGraph:edges', {TABLE_ATTRIBUTES => {DURABILITY => 'USE_DEFAULT'}, {NAME => 'f', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'SNAPPY', BLOCKCACHE => 'true', BLOCKSIZE => '131072'}

 

I attached hbase-config.xml and some server logs(for 2 hours).

 

Thanks.

 

TotalCompactionQueue.PNGHB_Home_Page.PNGMemory.PNGFlushQueue.PNG

Don't have an account?
Coming from Hortonworks? Activate your account here