Now I'm evaluating to HBase performance and strugging with compaction.
The queue of compaction does not increase until writing to 100[GB].
But after then, The queue continues to increase by writing data.
The compaction queue size becomes around 3000 per each nodes when my app wrote 1.2[TB]. (Writing speed is 0.5[MB/s], and there is no reading operation while writing)
The queue is decreasing by 1000 per day per node.
It takes 3 days to store 1.2[TB], and it takes 3 days to completed compaction...
My concern is
- I disabled major compaction. Does minor compaction takes such a long time? (or failed to disable it?)
- Can I shorten the compaction execution time with keeping writing speed?
- If I succeed in shortening, will it affect reading latency?
I changed following settings from default.
- hbase.hregion.majorcompaction : 0
- hbase.hstore.blockingWaitTime : 0
- hbase.hregion.memstore.block.multiplier : 10
- Specifiy pre-split key when create table
- The block size is 1[MB]
My HBase evaluation is
- region server : 3 nodes
- Software : windows, HDP-2.4.2, hbase-220.127.116.11.4.0.0-169
- Set 20[GB] memory to each region server
- Each row size is 6[MB]. column family number is 3, qualifier number is 18.
- The total row is 200K (so total data volume is 6[MB] * 200K = 1.2[TB])
- Writing performance : 0.5[MB/s]
Well, you have too many minor compactions. Why disabling major compactions? Or if you do, you should run major compactions manually from time to time. I'd enable major compactions, pre-split tables, salt keys if they are uniform, enable compression etc. Or, if your data is ready, you can do bulk import. Also, only 3 region servers is too little. Can you afford at least 6 with less memory, like 12 or 16G?
Thank you for quick reply.
Bulk import sounds good. I will try it.
The reason of disabling major compaction is I was concerned about the load when majro and minor compaction were executed at the same time.
And I'm preparing to getting aditional 3 PC. But it takes several month...
I think writing speed 0.5 [MB / s] with 3 nodes is slow.
Is this speed normal? Or is tuning difficult in my small PC environment?
This block size if specified in HColumnDescriptor. And HDFS block size default 64[MB].
Each row has 6[MB], So one block can keep 6rows. I think 6 is small..., Should I increase block size?
How many disks you have per node? Can you please check the region server logs about compaction related logging? Everytime a compaction runs and finishes, there should be some log saying how much data is compacted and how long it took etc. There seems to be something misconfigured in your environment.
I checked log about "CompactSplitThread". It finished in several seconds~minutes and it treats 10~100 MB files before the total size reached around 1[TB]. But after then it treats several GB file and it takes several 10 minues. So I thought threshold is set in 1[TB] about heavy compaction/split.
hbase.hregion.max.filesize causes "major compaction" as following site. But it was set 1[GB](default value).
I do not think that this setting is the cause...
But I have no other choise, I increased this value to 10[TB] and change policy to
ConstantSizeRegionSplitPolicy. I began to write record again.
I hope this will not affect bad because my table is pre-splitted. I'll report if it goes well.
Each node has one 7.2K rpm 3TB HDD.
And today 2 nodes finished compaction. But compaction queue size of 1 node increased from yesterday night...
I checked regionserver log.
3 type compaction(HStore, HRegion and CompactSplitThread) looks working every time.
It takes serveral ten minutes and it treats several GB file for each time.
I set pre-split to my table. But split looks working. Can it cause slowness?
I attached my region server log file. Could you give me some advice if you have time?
* I deleted "org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz]" from log file. Because it recorded several times every second.