Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Region server was exiting or aborted due to memstore size Errors

Region server was exiting or aborted due to memstore size Errors

Explorer

Hello Team,

 

I am facing issues with my region servers, We are seeing frequent failures.

Find below logs and suggest me with your valuable comments.

 

Spoiler

2019-10-30 14:35:08,532 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog: Failed close of WAL writer hdfs://nameservice1/hbase/WALs/region_server_hostname,60020,1558085878106/region_server_hostname%2C60020%2C1558085878106.null0.1572459736807, unflushedEntries=1
org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1869)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:954)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:728)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2037)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1926)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -3
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2033)
... 5 more
2019-10-30 14:35:08,535 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server region_server_hostname,60020,1558085878106: Failed log close in log roller
org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://nameservice1/hbase/WALs/region_server_hostname,60020,1558085878106/region_server_hostname%2C60020%2C1558085878106.null0.1572459736807, unflushedEntries=1
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:1004)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:728)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1869)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:954)
... 3 more
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2037)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1926)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -3
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2033)
... 5 more
2019-10-30 14:35:08,535 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [com.nielsen.engineering.asg.MatcherHBaseCoprocessorPB]

2019-10-30 14:35:08,671 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1144896
2019-10-30 14:35:08,677 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 17453136
2019-10-30 14:35:08,679 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 600
2019-10-30 14:35:08,685 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 58400
2019-10-30 14:35:08,686 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 26146216
2019-10-30 14:35:08,693 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 175752
2019-10-30 14:35:08,699 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 12884600
2019-10-30 14:35:08,703 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 76648
2019-10-30 14:35:08,720 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4231816
2019-10-30 14:35:08,727 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 576
2019-10-30 14:35:08,730 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5062392
2019-10-30 14:35:08,744 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2677936
2019-10-30 14:35:08,753 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2674992
2019-10-30 14:35:08,757 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 37561944
2019-10-30 14:35:08,760 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3264
2019-10-30 14:35:08,765 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 192
2019-10-30 14:35:08,767 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 38161048
2019-10-30 14:35:08,781 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2645184
2019-10-30 14:35:08,791 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 523664
2019-10-30 14:35:08,804 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 116112
2019-10-30 14:35:08,807 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 6306416
2019-10-30 14:35:08,814 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2187848
2019-10-30 14:35:08,819 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1068856
2019-10-30 14:35:08,826 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 161600
2019-10-30 14:35:08,839 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 27230344
2019-10-30 14:35:08,847 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 144216
2019-10-30 14:35:08,850 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 311120
2019-10-30 14:35:08,853 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2454376
2019-10-30 14:35:08,860 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 484472
2019-10-30 14:35:08,864 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2150040
2019-10-30 14:35:08,873 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5324592
2019-10-30 14:35:08,875 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 12694896
2019-10-30 14:35:08,879 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11750056
2019-10-30 14:35:08,883 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 631120
2019-10-30 14:35:08,893 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 81144
2019-10-30 14:35:08,894 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1861896
2019-10-30 14:35:08,895 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11443144
2019-10-30 14:35:08,902 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 145224
2019-10-30 14:35:08,904 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5860216
2019-10-30 14:35:08,909 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 578312
2019-10-30 14:35:08,911 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 211896
2019-10-30 14:35:08,917 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 180008
2019-10-30 14:35:08,919 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 26812296
2019-10-30 14:35:08,931 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5652848
2019-10-30 14:35:08,933 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 44270032
2019-10-30 14:35:08,938 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11262456
2019-10-30 14:35:08,940 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2463392
2019-10-30 14:35:08,946 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 25156664
2019-10-30 14:35:08,950 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 689816
2019-10-30 14:35:08,964 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1300696
2019-10-30 14:35:08,966 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 20891544
2019-10-30 14:35:08,967 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 18861840
2019-10-30 14:35:08,972 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 475864
2019-10-30 14:35:08,979 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 968944
2019-10-30 14:35:08,982 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1790688
2019-10-30 14:35:08,989 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2243328
2019-10-30 14:35:08,991 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8916640
2019-10-30 14:35:08,994 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3552
2019-10-30 14:35:08,997 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 13679296
2019-10-30 14:35:09,003 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 585824
2019-10-30 14:35:09,005 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3884792
2019-10-30 14:35:09,011 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 153176
2019-10-30 14:35:09,013 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 80299072
2019-10-30 14:35:09,024 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 7077744
2019-10-30 14:35:09,024 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 54805504
2019-10-30 14:35:09,030 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 12468392
2019-10-30 14:35:09,039 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 56855448
2019-10-30 14:35:09,046 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 65344
2019-10-30 14:35:09,052 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 49448344
2019-10-30 14:35:09,056 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8822616
2019-10-30 14:35:09,064 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 7830856
2019-10-30 14:35:09,071 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 20791080
2019-10-30 14:35:09,084 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 43269992
2019-10-30 14:35:09,093 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 24443664
2019-10-30 14:35:09,105 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 10384
2019-10-30 14:35:09,109 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3730232
2019-10-30 14:35:09,112 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 78640
2019-10-30 14:35:09,117 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 99272
2019-10-30 14:35:09,129 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 85928
2019-10-30 14:35:09,137 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 33360
2019-10-30 14:35:09,138 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1427656
2019-10-30 14:35:09,153 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 815488
2019-10-30 14:35:09,162 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2647576
2019-10-30 14:35:09,168 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 28870152
2019-10-30 14:35:09,173 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 57004488
2019-10-30 14:35:09,188 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8407696
2019-10-30 14:35:09,188 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2447016
2019-10-30 14:35:09,194 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4367608
2019-10-30 14:35:09,202 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1817736
2019-10-30 14:35:09,213 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3072800
2019-10-30 14:35:09,216 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 10939904
2019-10-30 14:35:09,218 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 12518256
2019-10-30 14:35:09,225 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 21037648
2019-10-30 14:35:09,230 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2881992
2019-10-30 14:35:09,235 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 13761728
2019-10-30 14:35:09,242 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 20209456
2019-10-30 14:35:09,243 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 80600
2019-10-30 14:35:09,257 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 25041480
2019-10-30 14:35:09,263 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4688504
2019-10-30 14:35:09,264 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8553240
2019-10-30 14:35:09,270 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 964160
2019-10-30 14:35:09,285 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 462576
2019-10-30 14:35:09,286 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 3590392
2019-10-30 14:35:09,296 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5370224
2019-10-30 14:35:09,299 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 68315704
2019-10-30 14:35:09,300 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 40664
2019-10-30 14:35:09,311 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1316704
2019-10-30 14:35:09,314 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 277568
2019-10-30 14:35:09,315 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 5363048
2019-10-30 14:35:09,320 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 105192
2019-10-30 14:35:09,322 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 49936312
2019-10-30 14:35:09,327 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 155584
2019-10-30 14:35:09,329 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 135576
2019-10-30 14:35:09,334 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 111520
2019-10-30 14:35:09,342 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1771184
2019-10-30 14:35:09,346 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4006968
2019-10-30 14:35:09,347 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 81198280
2019-10-30 14:35:09,359 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1330504
2019-10-30 14:35:09,365 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 8752144
2019-10-30 14:35:09,366 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 576
2019-10-30 14:35:09,367 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 4723464
2019-10-30 14:35:09,373 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 192
2019-10-30 14:35:09,384 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 831128
2019-10-30 14:35:09,387 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 165912
2019-10-30 14:35:09,391 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 168560
2019-10-30 14:35:09,392 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 9395224
2019-10-30 14:35:09,395 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11874440
2019-10-30 14:35:09,402 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 137216
2019-10-30 14:35:09,406 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 2987056
2019-10-30 14:35:09,407 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 14720
2019-10-30 14:35:09,416 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 275448
2019-10-30 14:35:09,420 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 7800312
2019-10-30 14:35:09,433 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 13289768
2019-10-30 14:35:09,435 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 46125672
2019-10-30 14:35:09,438 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 1234696
2019-10-30 14:35:09,440 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 11315448
2019-10-30 14:35:09,440 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 164808
2019-10-30 14:35:09,519 ERROR org.apache.hadoop.hbase.regionserver.HRegion: Memstore size is 68941424
2019-10-30 14:35:10,160 ERROR org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine: Region server exiting
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2540)

 

 

My configurations are like below,

hbase.hregion.memstore.flush.size = 128 MB
hbase.hregion.preclose.flush.size= 5 MB
hbase.hregion.memstore.block.multiplier = 2
hbase.hstore.compactionThreshold = 32
Maximum Size of All Memstores in RegionServer = 0.25  (As per my knowledge we have to keep this as 40% of heap which means 0.4)
 
Low Watermark for Memstore Flush = 0.15 (As per my knowledge we have to keep this as 35% of heap which is 0.35)
Each RegionServer RAM is 256 MB with 56 cores.
 
 
Please can some one help me on this issue.
Thanks in advance.
 
Best Regards,
Vinod
2 REPLIES 2
Highlighted

Re: Region server was exiting or aborted due to memstore size Errors

Explorer

Hello,

 

Any help from any one ? or Any suggestions ?

 

Thanks,

Vinod

Highlighted

Re: Region server was exiting or aborted due to memstore size Errors

Explorer

Hello Team,

 

Any help should be appreciated.

Thanks in advance ..!

 

Regards,

Vinod 

 

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here