Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hbase region servers are going to dead how to resolve the issueand also regions are failed to open

avatar
Explorer

[team_hdfs9@hadoop5 hbase]$ tail -f hbase-hbase-regionserver-hadoop5.log at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5978) ... 10 more Caused by: java.lang.IllegalArgumentException: Need table descriptor at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:654) at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631) ... 14 more 2017-04-25 04:17:09,197 INFO [RS_OPEN_REGION-hadoop5:16020-2] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => a97d8ea1c2947d5e13eb95231d19d924, NAME => 'CampaignSMSTransaction,cccccccc,1489751766841.a97d8ea1c2947d5e13eb95231d19d924.', STARTKEY => 'cccccccc', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:17:09,197 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioning a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:17:09,204 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioned node a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:18:41,384 DEBUG [RpcServer.reader=2,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44584 because read count=-1. Number of active connections: 2 2017-04-25 04:19:09,176 DEBUG [RpcServer.reader=1,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44583 because read count=-1. Number of active connections: 1 failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:21:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2017-04-25 04:26:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 ^C [team_hdfs9@hadoop5 hbase]$ tail -f hbase-hbase-regionserver-hadoop5.log at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:654) at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631) ... 14 more 2017-04-25 04:17:09,197 INFO [RS_OPEN_REGION-hadoop5:16020-2] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => a97d8ea1c2947d5e13eb95231d19d924, NAME => 'CampaignSMSTransaction,cccccccc,1489751766841.a97d8ea1c2947d5e13eb95231d19d924.', STARTKEY => 'cccccccc', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:17:09,197 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioning a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:17:09,204 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioned node a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:18:41,384 DEBUG [RpcServer.reader=2,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44584 because read count=-1. Number of active connections: 2 2017-04-25 04:19:09,176 DEBUG [RpcServer.reader=1,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44583 because read count=-1. Number of active connections: 1 2017-04-25 04:21:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2017-04-25 04:26:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0

1 REPLY 1

avatar
Super Guru

Use the HBCK tool to identify corruption of HBase data in HDFS

$ hbase hbck

You should include a full log file next time, but it would appear that the data in HDFS is corrupt "Caused by: java.lang.IllegalArgumentException: Need table descriptor". The table descriptor is stored in a file in HDFS.