Minor compaction extermly slow

[#1]

Hello.

Now I'm evaluating to HBase performance and strugging with compaction.

The queue of compaction does not increase until writing to 100[GB].

But after then, The queue continues to increase by writing data.

The compaction queue size becomes around 3000 per each nodes when my app wrote 1.2[TB]. (Writing speed is 0.5[MB/s], and there is no reading operation while writing)

The queue is decreasing by 1000 per day per node.

It takes 3 days to store 1.2[TB], and it takes 3 days to completed compaction...

My concern is

- I disabled major compaction. Does minor compaction takes such a long time? (or failed to disable it?)

- Can I shorten the compaction execution time with keeping writing speed?

- If I succeed in shortening, will it affect reading latency?

I changed following settings from default.

- hbase.hregion.majorcompaction : 0

- hbase.hstore.blockingWaitTime : 0

- hbase.hregion.memstore.block.multiplier : 10

- Specifiy pre-split key when create table

- The block size is 1[MB]

My HBase evaluation is

- region server : 3 nodes

- Software : windows, HDP-2.4.2, hbase-1.1.2.2.4.0.0-169

- Set 20[GB] memory to each region server

- Each row size is 6[MB]. column family number is 3, qualifier number is 18.

- The total row is 200K (so total data volume is 6[MB] * 200K = 1.2[TB])

- Writing performance : 0.5[MB/s]

[#2]

Well, you have too many minor compactions. Why disabling major compactions? Or if you do, you should run major compactions manually from time to time. I'd enable major compactions, pre-split tables, salt keys if they are uniform, enable compression etc. Or, if your data is ready, you can do bulk import. Also, only 3 region servers is too little. Can you afford at least 6 with less memory, like 12 or 16G?

[#3]

Thank you for quick reply.

Bulk import sounds good. I will try it.

The reason of disabling major compaction is I was concerned about the load when majro and minor compaction were executed at the same time.

And I'm preparing to getting aditional 3 PC. But it takes several month...

I think writing speed 0.5 [MB / s] with 3 nodes is slow.

Is this speed normal? Or is tuning difficult in my small PC environment?

[#4]

You made the HDFS block size 1MB? That seems excessively small.

[#5]

This block size if specified in HColumnDescriptor. And HDFS block size default 64[MB].

Each row has 6[MB], So one block can keep 6rows. I think 6 is small..., Should I increase block size?

[#6]

How many disks you have per node? Can you please check the region server logs about compaction related logging? Everytime a compaction runs and finishes, there should be some log saying how much data is compacted and how long it took etc. There seems to be something misconfigured in your environment.

[#7]

I checked log about "CompactSplitThread". It finished in several seconds~minutes and it treats 10~100 MB files before the total size reached around 1[TB]. But after then it treats several GB file and it takes several 10 minues. So I thought threshold is set in 1[TB] about heavy compaction/split.

hbase.hregion.max.filesize causes "major compaction" as following site. But it was set 1[GB](default value).

I do not think that this setting is the cause...

https://jp.hortonworks.com/blog/apache-hbase-region-splitting-and-merging/

But I have no other choise, I increased this value to 10[TB] and change policy to

ConstantSizeRegionSplitPolicy. I began to write record again.

I hope this will not affect bad because my table is pre-splitted. I'll report if it goes well.

[#8]

hbase-regionserver-node003-extract.zip

Each node has one 7.2K rpm 3TB HDD.

And today 2 nodes finished compaction. But compaction queue size of 1 node increased from yesterday night...

I checked regionserver log.

3 type compaction(HStore, HRegion and CompactSplitThread) looks working every time.

It takes serveral ten minutes and it treats several GB file for each time.

I set pre-split to my table. But split looks working. Can it cause slowness?

I attached my region server log file. Could you give me some advice if you have time?

* I deleted "org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz]" from log file. Because it recorded several times every second.