Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Minor compaction extermly slow

avatar
Explorer

Hello.

Now I'm evaluating to HBase performance and strugging with compaction.

The queue of compaction does not increase until writing to 100[GB].

But after then, The queue continues to increase by writing data.

The compaction queue size becomes around 3000 per each nodes when my app wrote 1.2[TB]. (Writing speed is 0.5[MB/s], and there is no reading operation while writing)

The queue is decreasing by 1000 per day per node.

It takes 3 days to store 1.2[TB], and it takes 3 days to completed compaction...

My concern is

- I disabled major compaction. Does minor compaction takes such a long time? (or failed to disable it?)

- Can I shorten the compaction execution time with keeping writing speed?

- If I succeed in shortening, will it affect reading latency?

I changed following settings from default.

- hbase.hregion.majorcompaction : 0

- hbase.hstore.blockingWaitTime : 0

- hbase.hregion.memstore.block.multiplier : 10

- Specifiy pre-split key when create table

- The block size is 1[MB]

My HBase evaluation is

- region server : 3 nodes

- Software : windows, HDP-2.4.2, hbase-1.1.2.2.4.0.0-169

- Set 20[GB] memory to each region server

- Each row size is 6[MB]. column family number is 3, qualifier number is 18.

- The total row is 200K (so total data volume is 6[MB] * 200K = 1.2[TB])

- Writing performance : 0.5[MB/s]

7 REPLIES 7

avatar
Master Guru

Well, you have too many minor compactions. Why disabling major compactions? Or if you do, you should run major compactions manually from time to time. I'd enable major compactions, pre-split tables, salt keys if they are uniform, enable compression etc. Or, if your data is ready, you can do bulk import. Also, only 3 region servers is too little. Can you afford at least 6 with less memory, like 12 or 16G?

avatar
Explorer

Thank you for quick reply.

Bulk import sounds good. I will try it.

The reason of disabling major compaction is I was concerned about the load when majro and minor compaction were executed at the same time.

And I'm preparing to getting aditional 3 PC. But it takes several month...

I think writing speed 0.5 [MB / s] with 3 nodes is slow.

Is this speed normal? Or is tuning difficult in my small PC environment?

avatar
Super Guru

You made the HDFS block size 1MB? That seems excessively small.

avatar
Explorer

This block size if specified in HColumnDescriptor. And HDFS block size default 64[MB].

Each row has 6[MB], So one block can keep 6rows. I think 6 is small..., Should I increase block size?

avatar
Guru

How many disks you have per node? Can you please check the region server logs about compaction related logging? Everytime a compaction runs and finishes, there should be some log saying how much data is compacted and how long it took etc. There seems to be something misconfigured in your environment.

avatar
Explorer

I checked log about "CompactSplitThread". It finished in several seconds~minutes and it treats 10~100 MB files before the total size reached around 1[TB]. But after then it treats several GB file and it takes several 10 minues. So I thought threshold is set in 1[TB] about heavy compaction/split.

hbase.hregion.max.filesize causes "major compaction" as following site. But it was set 1[GB](default value).

I do not think that this setting is the cause...

https://jp.hortonworks.com/blog/apache-hbase-region-splitting-and-merging/

But I have no other choise, I increased this value to 10[TB] and change policy to

ConstantSizeRegionSplitPolicy. I began to write record again.

I hope this will not affect bad because my table is pre-splitted. I'll report if it goes well.

avatar
Explorer

hbase-regionserver-node003-extract.zip

Each node has one 7.2K rpm 3TB HDD.

And today 2 nodes finished compaction. But compaction queue size of 1 node increased from yesterday night...

I checked regionserver log.

3 type compaction(HStore, HRegion and CompactSplitThread) looks working every time.

It takes serveral ten minutes and it treats several GB file for each time.

I set pre-split to my table. But split looks working. Can it cause slowness?

I attached my region server log file. Could you give me some advice if you have time?

* I deleted "org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz]" from log file. Because it recorded several times every second.