Hadoop HBase nonstop Memstore flushing and compactions

[#1]

Hello and thanks for reading 

I have an issue with hadoop hbase 

while trying to find the issue I can see below memstore flush logs for all tables and its non stop

MemstoreFlusherChore requesting flush of table 1. because K has an old edit so flush to free WALs after random delay 65889ms

022-10-29 12:22:13,176 INFO  [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of table 1. because K has an old edit so flush to free WALs after random delay 65889ms
2022-10-29 12:22:13,177 INFO  [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of tabl 2 . because K has an old edit so flush to free WALs after random delay 181952ms
2022-10-29 12:22:13,177 INFO  [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of table 3 . because K has an old edit so flush to free WALs after random delay 150462ms
2022-10-29 12:22:13,177 INFO  [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of table 4 . because K has an old edit so flush to free WALs after random delay 161958ms

This error seems complex because no documentation about error  

below is random logs from same node 

2022-10-29 12:21:46,170 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~6.68 MB/7002393, heapSize ~19.57 MB/20517200, currentSize=0 B/0 for ad1bd353d9ae0c52e30a935c5d06ecfa in 335ms, sequenceid=25141679, compaction requested=true

2022-10-29 12:22:02,679 INFO  [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, dataSize=2.79 MB heapSize=7.22 MB

Tasks:
===========================================================
Task: Compacting K in table .
Status: RUNNING:Compacting store K
Running for 11449s

Task: Compacting K in region.
Status: RUNNING:Compacting store K
Running for 168s

Task: Compacting K in region.
Status: RUNNING:Compacting store K
Running for 159s

Task: Compacting K in region.
Status: RUNNING:Compacting store K
Running for 72s

RS Queue:
===========================================================
Compaction/Split Queue summary: compactionQueue=(longCompactions=111975:shortCompactions=35811), splitQueue=0
Compaction/Split Queue dump:
  LargeCompation Queue:

Below is the worker node data 

 

 

 

 

I tried to find any configuration for WAL, but seems it has been depricated

HBase Version   2.0.0.3.0.1.0-187, revision=e9fcf450949102de5069b257a6dee469b8f5aab3

Hadoop Version 3.1.1.3.0.1.0-187, revision=2820e4d6fc7ec31ac42187083ed5933c823e9784

[#2 · Accepted solution]

Hello @TheFixer 

Thanks for using Cloudera Community. To your queries, Please find the details below:

(I) Concerning below log tracing, Each Column Family has their own MemStore & HBase ensure the Flush happens before an Edit is too old. A WAL can't be archived unless the corresponding entries from the WAL remain Un-flushed from the MemStore. These logging aren't a concern, yet PeriodicMemstoreFlusher working as designed.

MemstoreFlusherChore requesting flush of table 1. because K has an old edit so flush to free WALs after random delay 65889ms

 
(II) Concerning below log tracing, the same indicates the Flush has Completed by "MemStoreFlusher.1" (Defined by "hbase.hstore.flusher.count") of size ~6.68MB & the SequenceId indicates the Sequence ID associated with the LastEdit flushed. This Sequence ID is used to compare with Edits in WAL prior to archiving the WAL.

2022-10-29 12:21:46,170 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~6.68 MB/7002393, heapSize ~19.57 MB/20517200, currentSize=0 B/0 for ad1bd353d9ae0c52e30a935c5d06ecfa in 335ms, sequenceid=25141679, compaction requested=true

 (III) Finally, the CompactionQueue indicates the Size of Major (Long) & Minor (Short) Compaction.

Compaction/Split Queue summary: compactionQueue=(longCompactions=111975:shortCompactions=35811), splitQueue=0

All the above Logging are HBase Internals & shouldn't be a concern for Customer, unless Customer is observing any impact. Your Team should review (If not done yet) the below links for details around the Flush Internals & Compaction Algorithms:
(I) MemStore Flush Configuration
(II) Conditions For MemStore Flush
(III) Compaction Alogrithm

Kindly review & let us know if your Team have any queries.

Regards, Smarak

[#3]

Hello @TheFixer 

Kindly review & let us know if you have reviewed our Post dated 2022-11-07 & have any further queries concerning the queries around HBase RegionServer logging highlighted in the Post. 

Regards, Smarak

[#4]

Hello @TheFixer 

We hope your queries around the RegionServer log traces has been answered by us. As such, We shall mark the Post as Solved. If you have any further ask, Feel free to update the Post & we shall get back to you accordingly.

Regards, Smarak