Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hbase region servers are going to dead how to resolve the issueand also regions are failed to open

hbase region servers are going to dead how to resolve the issueand also regions are failed to open

New Contributor

[team_hdfs9@hadoop5 hbase]$ tail -f hbase-hbase-regionserver-hadoop5.log at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5978) ... 10 more Caused by: java.lang.IllegalArgumentException: Need table descriptor at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:654) at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631) ... 14 more 2017-04-25 04:17:09,197 INFO [RS_OPEN_REGION-hadoop5:16020-2] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => a97d8ea1c2947d5e13eb95231d19d924, NAME => 'CampaignSMSTransaction,cccccccc,1489751766841.a97d8ea1c2947d5e13eb95231d19d924.', STARTKEY => 'cccccccc', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:17:09,197 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioning a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:17:09,204 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioned node a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:18:41,384 DEBUG [RpcServer.reader=2,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44584 because read count=-1. Number of active connections: 2 2017-04-25 04:19:09,176 DEBUG [RpcServer.reader=1,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44583 because read count=-1. Number of active connections: 1 failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:21:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2017-04-25 04:26:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 ^C [team_hdfs9@hadoop5 hbase]$ tail -f hbase-hbase-regionserver-hadoop5.log at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:654) at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631) ... 14 more 2017-04-25 04:17:09,197 INFO [RS_OPEN_REGION-hadoop5:16020-2] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => a97d8ea1c2947d5e13eb95231d19d924, NAME => 'CampaignSMSTransaction,cccccccc,1489751766841.a97d8ea1c2947d5e13eb95231d19d924.', STARTKEY => 'cccccccc', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:17:09,197 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioning a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:17:09,204 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioned node a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:18:41,384 DEBUG [RpcServer.reader=2,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44584 because read count=-1. Number of active connections: 2 2017-04-25 04:19:09,176 DEBUG [RpcServer.reader=1,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44583 because read count=-1. Number of active connections: 1 2017-04-25 04:21:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2017-04-25 04:26:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0

1 REPLY 1

Re: hbase region servers are going to dead how to resolve the issueand also regions are failed to open

Use the HBCK tool to identify corruption of HBase data in HDFS

$ hbase hbck

You should include a full log file next time, but it would appear that the data in HDFS is corrupt "Caused by: java.lang.IllegalArgumentException: Need table descriptor". The table descriptor is stored in a file in HDFS.

Don't have an account?
Coming from Hortonworks? Activate your account here