Support Questions

Find answers, ask questions, and share your expertise

hbase region servers are going to dead how to resolve the issueand also regions are failed to open

avatar
Explorer

[team_hdfs9@hadoop5 hbase]$ tail -f hbase-hbase-regionserver-hadoop5.log at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5978) ... 10 more Caused by: java.lang.IllegalArgumentException: Need table descriptor at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:654) at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631) ... 14 more 2017-04-25 04:17:09,197 INFO [RS_OPEN_REGION-hadoop5:16020-2] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => a97d8ea1c2947d5e13eb95231d19d924, NAME => 'CampaignSMSTransaction,cccccccc,1489751766841.a97d8ea1c2947d5e13eb95231d19d924.', STARTKEY => 'cccccccc', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:17:09,197 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioning a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:17:09,204 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioned node a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:18:41,384 DEBUG [RpcServer.reader=2,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44584 because read count=-1. Number of active connections: 2 2017-04-25 04:19:09,176 DEBUG [RpcServer.reader=1,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44583 because read count=-1. Number of active connections: 1 failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:21:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2017-04-25 04:26:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 ^C [team_hdfs9@hadoop5 hbase]$ tail -f hbase-hbase-regionserver-hadoop5.log at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:654) at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:631) ... 14 more 2017-04-25 04:17:09,197 INFO [RS_OPEN_REGION-hadoop5:16020-2] coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => a97d8ea1c2947d5e13eb95231d19d924, NAME => 'CampaignSMSTransaction,cccccccc,1489751766841.a97d8ea1c2947d5e13eb95231d19d924.', STARTKEY => 'cccccccc', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 3 2017-04-25 04:17:09,197 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioning a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:17:09,204 DEBUG [RS_OPEN_REGION-hadoop5:16020-2] zookeeper.ZKAssign: regionserver:16020-0x15b9f5b78d301bc, quorum=hadoop7:2181,hadoop6:2181,hadoop5:2181, baseZNode=/hbase-unsecure Transitioned node a97d8ea1c2947d5e13eb95231d19d924 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2017-04-25 04:18:41,384 DEBUG [RpcServer.reader=2,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44584 because read count=-1. Number of active connections: 2 2017-04-25 04:19:09,176 DEBUG [RpcServer.reader=1,bindAddress=hadoop5,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.1.228:44583 because read count=-1. Number of active connections: 1 2017-04-25 04:21:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2017-04-25 04:26:10,682 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.14 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0

1 REPLY 1

avatar
Super Guru

Use the HBCK tool to identify corruption of HBase data in HDFS

$ hbase hbck

You should include a full log file next time, but it would appear that the data in HDFS is corrupt "Caused by: java.lang.IllegalArgumentException: Need table descriptor". The table descriptor is stored in a file in HDFS.