Member since
03-19-2020
1
Post
0
Kudos Received
0
Solutions
03-19-2020
02:10 AM
In our Hbase cluster, there is a never ending compaction queue, which is also getting larger day by day. We've changed and edited a lot of parameters, but nothing helped. This situtation is also affecting cluster performance and stability. For this specific problem, we performed a thorough reading on Hbase documentation, and did a lot of tests according to our inquiries, but did not get a satisfactory result. We are using Hbase as a graph store. On a daily basis, it grows by 1,5 TB/1,5 billion rows. The rows are relatively small size. And the operations are write-intensive. Our 17 region servers have 256 GB of memory installed on them and we have not observed any bottleneck on servers' resources. We are using multiple tables, but here are the definitions of tables, which are doing the most of the I/O. 1- 'myGraph:edgeIndices', {TABLE_ATTRIBUTES => {DURABILITY => 'USE_DEFAULT'}, {NAME => 'f', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'SNAPPY', BLOCKCACHE => 'true', BLOCKSIZE => '131072'} 2- 'myGraph:edges', {TABLE_ATTRIBUTES => {DURABILITY => 'USE_DEFAULT'}, {NAME => 'f', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'SNAPPY', BLOCKCACHE => 'true', BLOCKSIZE => '131072'} I attached hbase-config.xml and some server logs(for 2 hours). Thanks.
... View more
Labels:
- Labels:
-
Apache HBase