Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Region Servers are Frequently aborting due to ERROR org.apache.hadoop.hbase.regionserver.HRegion

Region Servers are Frequently aborting due to ERROR org.apache.hadoop.hbase.regionserver.HRegion

Rising Star

Hello Team,

 


We are facing frequent region server aborting in hbase and below is the error logs,


ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog
Failed close of WAL writer hdfs://nameservice1/hbase/WALs/regionserver74.enterprisenet.org,60020,1576226974029/regionserver74.enterprisenet.org%2C60020%2C1576226974029.null0.1585728806234, unflushedEntries=1
org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1869)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:954)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:728)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2037)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1926)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -3
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2033)
... 5 more


FATAL org.apache.hadoop.hbase.regionserver.HRegionServer
ABORTING region server regionserver74.enterprisenet.org,60020,1576226974029: Failed log close in log roller
org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://nameservice1/hbase/WALs/regionserver74.enterprisenet.org,60020,1576226974029/regionserver74.enterprisenet.org%2C60020%2C1576226974029.null0.1585728806234, unflushedEntries=1
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:1004)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:728)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1869)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:954)
... 3 more
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed offering sync
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2037)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1926)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: -3
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2033)
... 5 more


Apr 1, 4:41:04.707 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 1890784
Apr 1, 4:41:04.712 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 383952
Apr 1, 4:41:04.804 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 12854792
Apr 1, 4:41:04.828 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 192
Apr 1, 4:41:04.860 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 1001512
Apr 1, 4:41:04.884 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 187456
Apr 1, 4:41:04.920 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 17354880
Apr 1, 4:41:04.964 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 108344
Apr 1, 4:41:04.983 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 928648
Apr 1, 4:41:05.031 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 10125336
Apr 1, 4:41:05.151 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 7477760
Apr 1, 4:41:05.199 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 82640
Apr 1, 4:41:05.249 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 1609448
Apr 1, 4:41:05.269 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 13459968
Apr 1, 4:41:05.316 AM ERROR org.apache.hadoop.hbase.regionserver.HRegion
Memstore size is 6679936

 

Can some one please help me to fix this issue?

 

Best Regards,
Vinod

Don't have an account?
Coming from Hortonworks? Activate your account here