Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

Region server was exiting or aborted due to memstore size Errors

Rising Star

Hello Team,

 

I am facing issues with my region servers, We are seeing frequent failures.

Find below logs and suggest me with your valuable comments.

 

Spoiler

2019-10-30 14:35:08,532 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog: Failed close of WAL writer hdfs://nameservice1/hbase/WALs/region_server_hostname,60020,1558085878106/region_server_hostname%2C60020%2C1558085878106.null0.1572459736807, unflushedEntries=1
org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1869)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:954)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:728)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2037)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1926)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -3
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2033)
... 5 more
2019-10-30 14:35:08,535 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server region_server_hostname,60020,1558085878106: Failed log close in log roller
org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://nameservice1/hbase/WALs/region_server_hostname,60020,1558085878106/region_server_hostname%2C60020%2C1558085878106.null0.1572459736807, unflushedEntries=1
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:1004)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:728)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1869)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:954)
... 3 more
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2037)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1926)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -3
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2033)
... 5 more
2019-10-30 14:35:08,535 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [com.nielsen.engineering.asg.MatcherHBaseCoprocessorPB]

2019-10-30 14:35:08,671 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1144896
2019-10-30 14:35:08,677 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 17453136
2019-10-30 14:35:08,679 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 600
2019-10-30 14:35:08,685 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 58400
2019-10-30 14:35:08,686 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 26146216
2019-10-30 14:35:08,693 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 175752
2019-10-30 14:35:08,699 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 12884600
2019-10-30 14:35:08,703 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 76648
2019-10-30 14:35:08,720 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4231816
2019-10-30 14:35:08,727 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 576
2019-10-30 14:35:08,730 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5062392
2019-10-30 14:35:08,744 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2677936
2019-10-30 14:35:08,753 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2674992
2019-10-30 14:35:08,757 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 37561944
2019-10-30 14:35:08,760 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3264
2019-10-30 14:35:08,765 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 192
2019-10-30 14:35:08,767 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 38161048
2019-10-30 14:35:08,781 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2645184
2019-10-30 14:35:08,791 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 523664
2019-10-30 14:35:08,804 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 116112
2019-10-30 14:35:08,807 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 6306416
2019-10-30 14:35:08,814 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2187848
2019-10-30 14:35:08,819 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1068856
2019-10-30 14:35:08,826 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 161600
2019-10-30 14:35:08,839 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 27230344
2019-10-30 14:35:08,847 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 144216
2019-10-30 14:35:08,850 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 311120
2019-10-30 14:35:08,853 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2454376
2019-10-30 14:35:08,860 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 484472
2019-10-30 14:35:08,864 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2150040
2019-10-30 14:35:08,873 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5324592
2019-10-30 14:35:08,875 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 12694896
2019-10-30 14:35:08,879 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11750056
2019-10-30 14:35:08,883 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 631120
2019-10-30 14:35:08,893 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 81144
2019-10-30 14:35:08,894 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1861896
2019-10-30 14:35:08,895 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11443144
2019-10-30 14:35:08,902 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 145224
2019-10-30 14:35:08,904 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5860216
2019-10-30 14:35:08,909 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 578312
2019-10-30 14:35:08,911 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 211896
2019-10-30 14:35:08,917 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 180008
2019-10-30 14:35:08,919 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 26812296
2019-10-30 14:35:08,931 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5652848
2019-10-30 14:35:08,933 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 44270032
2019-10-30 14:35:08,938 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11262456
2019-10-30 14:35:08,940 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2463392
2019-10-30 14:35:08,946 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 25156664
2019-10-30 14:35:08,950 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 689816
2019-10-30 14:35:08,964 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1300696
2019-10-30 14:35:08,966 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 20891544
2019-10-30 14:35:08,967 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 18861840
2019-10-30 14:35:08,972 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 475864
2019-10-30 14:35:08,979 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 968944
2019-10-30 14:35:08,982 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1790688
2019-10-30 14:35:08,989 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2243328
2019-10-30 14:35:08,991 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8916640
2019-10-30 14:35:08,994 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3552
2019-10-30 14:35:08,997 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 13679296
2019-10-30 14:35:09,003 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 585824
2019-10-30 14:35:09,005 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3884792
2019-10-30 14:35:09,011 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 153176
2019-10-30 14:35:09,013 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 80299072
2019-10-30 14:35:09,024 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 7077744
2019-10-30 14:35:09,024 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 54805504
2019-10-30 14:35:09,030 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 12468392
2019-10-30 14:35:09,039 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 56855448
2019-10-30 14:35:09,046 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 65344
2019-10-30 14:35:09,052 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 49448344
2019-10-30 14:35:09,056 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8822616
2019-10-30 14:35:09,064 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 7830856
2019-10-30 14:35:09,071 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 20791080
2019-10-30 14:35:09,084 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 43269992
2019-10-30 14:35:09,093 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 24443664
2019-10-30 14:35:09,105 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 10384
2019-10-30 14:35:09,109 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3730232
2019-10-30 14:35:09,112 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 78640
2019-10-30 14:35:09,117 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 99272
2019-10-30 14:35:09,129 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 85928
2019-10-30 14:35:09,137 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 33360
2019-10-30 14:35:09,138 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1427656
2019-10-30 14:35:09,153 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 815488
2019-10-30 14:35:09,162 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2647576
2019-10-30 14:35:09,168 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 28870152
2019-10-30 14:35:09,173 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 57004488
2019-10-30 14:35:09,188 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8407696
2019-10-30 14:35:09,188 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2447016
2019-10-30 14:35:09,194 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4367608
2019-10-30 14:35:09,202 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1817736
2019-10-30 14:35:09,213 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3072800
2019-10-30 14:35:09,216 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 10939904
2019-10-30 14:35:09,218 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 12518256
2019-10-30 14:35:09,225 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 21037648
2019-10-30 14:35:09,230 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2881992
2019-10-30 14:35:09,235 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 13761728
2019-10-30 14:35:09,242 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 20209456
2019-10-30 14:35:09,243 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 80600
2019-10-30 14:35:09,257 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 25041480
2019-10-30 14:35:09,263 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4688504
2019-10-30 14:35:09,264 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8553240
2019-10-30 14:35:09,270 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 964160
2019-10-30 14:35:09,285 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 462576
2019-10-30 14:35:09,286 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3590392
2019-10-30 14:35:09,296 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5370224
2019-10-30 14:35:09,299 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 68315704
2019-10-30 14:35:09,300 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 40664
2019-10-30 14:35:09,311 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1316704
2019-10-30 14:35:09,314 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 277568
2019-10-30 14:35:09,315 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5363048
2019-10-30 14:35:09,320 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 105192
2019-10-30 14:35:09,322 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 49936312
2019-10-30 14:35:09,327 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 155584
2019-10-30 14:35:09,329 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 135576
2019-10-30 14:35:09,334 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 111520
2019-10-30 14:35:09,342 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1771184
2019-10-30 14:35:09,346 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4006968
2019-10-30 14:35:09,347 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 81198280
2019-10-30 14:35:09,359 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1330504
2019-10-30 14:35:09,365 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8752144
2019-10-30 14:35:09,366 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 576
2019-10-30 14:35:09,367 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4723464
2019-10-30 14:35:09,373 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 192
2019-10-30 14:35:09,384 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 831128
2019-10-30 14:35:09,387 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 165912
2019-10-30 14:35:09,391 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 168560
2019-10-30 14:35:09,392 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 9395224
2019-10-30 14:35:09,395 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11874440
2019-10-30 14:35:09,402 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 137216
2019-10-30 14:35:09,406 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2987056
2019-10-30 14:35:09,407 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 14720
2019-10-30 14:35:09,416 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 275448
2019-10-30 14:35:09,420 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 7800312
2019-10-30 14:35:09,433 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 13289768
2019-10-30 14:35:09,435 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 46125672
2019-10-30 14:35:09,438 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1234696
2019-10-30 14:35:09,440 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11315448
2019-10-30 14:35:09,440 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 164808
2019-10-30 14:35:09,519 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 68941424
2019-10-30 14:35:10,160 ERROR org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine: Region server exiting
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2540)

 

 

My configurations are like below,

hbase.hregion.memstore.flush.size = 128 MB
hbase.hregion.preclose.flush.size= 5 MB
hbase.hregion.memstore.block.multiplier = 2
hbase.hstore.compactionThreshold = 32
Maximum Size of All Memstores in RegionServer = 0.25  (As per my knowledge we have to keep this as 40% of heap which means 0.4)
 
Low Watermark for Memstore Flush = 0.15 (As per my knowledge we have to keep this as 35% of heap which is 0.35)
Each RegionServer RAM is 256 MB with 56 cores.
 
 
Please can some one help me on this issue.
Thanks in advance.
 
Best Regards,
Vinod
2 REPLIES 2

Rising Star

Hello,

 

Any help from any one ? or Any suggestions ?

 

Thanks,

Vinod

Rising Star

Hello Team,

 

Any help should be appreciated.

Thanks in advance ..!

 

Regards,

Vinod