Support Questions
Find answers, ask questions, and share your expertise

Too frequent major compaction

Highlighted

Too frequent major compaction

Contributor

I have set major compaction frequency to 3 days.

	hbase.offpeak.start.hour = 19
	hbase.offpeak.end.hour = 1

However I see in RS metrics, for a particular table number of total major compaction is 22 ("Namespace_default_table_user_states_metric_majorCompactionTime_num_ops": 22) and the cluster was last restarted 14 days ago. So this is >1 major compaction a day.

Also in logs I see many major compaction running on single file. Despite it's set for 3 to 10 files. ( see the first line below)

2017-06-08 02:32:07,377 INFO  [StoreOpener-1f254a063af7b15509846ff27ef9b5a8-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 259200000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6


2017-06-08 00:28:36,581 INFO  [regionserver/<host>/10.10.210.195:16020-shortCompactions-1495621699452] regionserver.HStore: Completed major compaction of 1 (all) file(s) in f of events,voodoo.com:ringo:call_log:1433665626397:r116qjoprsjhppq1,1435904225768.c901139ea575133398f86ceb4dc253fa. into f275a4737fe1485e9058532a2226d401(size=13.7 M), total size for store is 13.7 M. This selection was in queue for 0sec, and took 2sec to execute.2017-06-08 

06:02:22,176 INFO  [regionserver/<host>/10.10.210.195:16020-shortCompactions-1495621699452] regionserver.HStore: Completed major compaction of 1 (all) file(s) in f of events,voodoo.com:ringo:call_log:1458694092743:7azkyvazxry7rxkr,1461000798058.5703747043306593388ed83c0d245772. into 168f131831244c9f8aafdcc9105e37dc(size=304.3 M), total size for store is 304.3 M. This selection was in queue for 0sec, and took 27sec to execute.

2017-06-08 06:03:55,214 INFO  [regionserver/<host>/10.10.210.195:16020-shortCompactions-1495621699452] regionserver.HStore: Completed major compaction of 1 (all) file(s) in f of events,voodoo.com:ringo:connectivity_log:1464497220532:y66a0afayoava6gn,1466007331891.d89b88bd976aab7e229116fcf4ac2fb0. into e7f6451d5f2a40a59795285c2d5c8feb(size=1.0 G), total size for store is 1.0 G. This selection was in queue for 28sec, and took 1mins, 32sec to execute.

This log line I find several times in a day and it's not limited to off-peak hours. I do not think frequency of major compaction should be this high. Or maybe the log line is suggesting something else and not major compaction.

hbase.hregion.max.filesize is 20G

hbase.hstore.compaction.max is 10

Any help with figuring out what is happening is appreciated. Please let me know if any more information is required.

Thank you!

2 REPLIES 2

Re: Too frequent major compaction

Contributor

Hi Sanket,


Did you solve this problem?

I am encountring the same problem.. many Major compaction the same day in the same Column Family and the same server?


Thanke

Helmi KHALIFA

Highlighted

Re: Too frequent major compaction

New Contributor

same issue here with default configuration (once in 7 days).

any suggestions?

 

2021-01-11 03:27:19,028 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 4 (all) file(s) in Info of CallLogs,\xE9\x7F\x9EJ\x10\x06L\xF7\x9A\xBF+\xCD\xA8\xB7\x9D\xBB,1608101963434.f0d1a5f4e816118ac167fe9730258102. into a54298ecc9594f9aa0cf6657a795bb54(size=6.0 G), total size for store is 6.0 G. This selection was in queue for 0sec, and took 56sec to execute.
2021-01-11 03:55:46,943 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 5 (all) file(s) in Info of CallLogs,M}\xC4;,1609043919090.8c704f7385c3c3b07bc3aa4be1adc577. into 1d3f086c88bc4972a2e550dd093e9824(size=5.7 G), total size for store is 5.7 G. This selection was in queue for 0sec, and took 1mins, 29sec to execute.
2021-01-11 04:24:56,056 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 4 (all) file(s) in Info of CallLogs,.\x81\xC6\x99e1K\x00\xAE\xB3@\x14g \x0Av,1608158031158.4998d8db979dfea2751136bf1767fb1b. into 4b1c6db0ed5d440d9adb58bf00109b57(size=5.6 G), total size for store is 5.6 G. This selection was in queue for 0sec, and took 1mins, 33sec to execute.
2021-01-11 05:36:34,562 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 5 (all) file(s) in Info of CallLogs,\x19~A\x8F\xD3^G\xFB\xB5!.\x8C6\xCB\xC7t,1607673667302.6c08c1a2f5648c5f190bc378f628a838. into 71d8c8268fcc46d4ad1be29a6c6ce880(size=5.9 G), total size for store is 5.9 G. This selection was in queue for 0sec, and took 1mins, 42sec to execute.
2021-01-11 05:38:13,268 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 4 (all) file(s) in Info of CallLogs,\xB9~\x8EX\xA5cH\xBE\x94g\xFF\xB76\xD6\x80/,1608131740376.47868b4d2475ef1fef1f23fea51b2e0f. into 21e524b8483047b7a6529ff20ea56602(size=5.9 G), total size for store is 5.9 G. This selection was in queue for 0sec, and took 1mins, 17sec to execute.
2021-01-11 07:11:21,277 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 3 (all) file(s) in Info of CallLogs,\x9E\x7F\xD9\xADe\x81H\x8C\x8E\x80\x87)\xE0G\xD7\xFE,1608533709336.c152e8f046c8f75ca265c1fc9c742909. into f4dc8f12dcee4f118acc779ae000ff6b(size=5.8 G), total size for store is 5.8 G. This selection was in queue for 0sec, and took 1mins, 19sec to execute.
2021-01-11 08:51:46,548 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 5 (all) file(s) in Info of CallLogs,\x7F\x81\x01M8{O\x92\x9C\xC2J\x01\xB7r8\xF4,1608529936723.34e0ffe299134cd8ad22145ae2314d3e. into 246537faa92741ea865437eebf2e1e9a(size=6.0 G), total size for store is 6.0 G. This selection was in queue for 0sec, and took 1mins, 29sec to execute.