Created 08-08-2016 02:22 PM
Unable to scan hbase table. getting following error-
how to recover table.
scan 'CUTOFF8'-
ERROR: org.apache.hadoop.hbase.NotServingRegionException: Region CUTOFF8,,1465897349742.2077c5dfbfb97d67f09120e4b9cdc15a. is not online on data1.corp.mirrorplus.com,16020,1470665536454 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2898) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:947) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2235) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745)
hbase master log-
2016-08-08 09:04:45,112 WARN [master1.corp.mirrorplus.com,16000,1470404494921_ChoreService_1] hdfs.DFSClient: DFS chooseDataNode: got # 3 IOException, will wait for 12866.800703353128 msec. 2016-08-08 09:04:57,979 WARN [master1.corp.mirrorplus.com,16000,1470404494921_ChoreService_1] hdfs.DFSClient: Could not obtain block: BP-838165258-10.1.1.94-1459246457024:blk_1073781564_40751 file=/apps/hbase/data1/.hbase-snapshot/S3LINKS-AIM-SNAPSHOT-NEW/.snapshotinfo No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException 2016-08-08 09:04:57,979 WARN [master1.corp.mirrorplus.com,16000,1470404494921_ChoreService_1] hdfs.DFSClient: Could not obtain block: BP-838165258-10.1.1.94-1459246457024:blk_1073781564_40751 file=/apps/hbase/data1/.hbase-snapshot/S3LINKS-AIM-SNAPSHOT-NEW/.snapshotinfo No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException 2016-08-08 09:04:57,979 WARN [master1.corp.mirrorplus.com,16000,1470404494921_ChoreService_1] hdfs.DFSClient: DFS Read org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-838165258-10.1.1.94-1459246457024:blk_1073781564_40751 file=/apps/hbase/data1/.hbase-snapshot/S3LINKS-AIM-SNAPSHOT-NEW/.snapshotinfo at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:945) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:604) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:844) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:896) at java.io.DataInputStream.read(DataInputStream.java:100) at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:737) at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701) at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99) at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$SnapshotDescription.<init>(HBaseProtos.java:10616) at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$SnapshotDescription.<init>(HBaseProtos.java:10580) at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$SnapshotDescription$1.parsePartialFrom(HBaseProtos.java:10694) at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$SnapshotDescription$1.parsePartialFrom(HBaseProtos.java:10689) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$SnapshotDescription.parseFrom(HBaseProtos.java:11177) at org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.readSnapshotInfo(SnapshotDescriptionUtils.java:307) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.getHFileNames(SnapshotReferenceUtil.java:328) at org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner$1.filesUnderSnapshot(SnapshotHFileCleaner.java:85) at org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.refreshCache(SnapshotFileCache.java:281) at org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.getUnreferencedFiles(SnapshotFileCache.java:187) at org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:62) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Created 08-08-2016 02:23 PM
A few things:
It looks like your HDFS is not healthy. Can you run `hdfs fsck` on the file "/apps/hbase/data1/.hbase-snapshot/S3LINKS-AIM-SNAPSHOT-NEW/.snapshotinfo"?
Regarding the offline region "CUTOFF8,,1465897349742.2077c5dfbfb97d67f09120e4b9cdc15a.", you can check the RegionServer logs on data1.corp.mirrorplus.com. Also, you can try to grep the HBase master log for "2077c5dfbfb97d67f09120e4b9cdc15a", looking for the last "OPEN" location. The HBase master UI might also tell you if this region is stuck in transition (RIT). If so, `hbase hbck` should be able to help you.
Created 08-08-2016 02:23 PM
A few things:
It looks like your HDFS is not healthy. Can you run `hdfs fsck` on the file "/apps/hbase/data1/.hbase-snapshot/S3LINKS-AIM-SNAPSHOT-NEW/.snapshotinfo"?
Regarding the offline region "CUTOFF8,,1465897349742.2077c5dfbfb97d67f09120e4b9cdc15a.", you can check the RegionServer logs on data1.corp.mirrorplus.com. Also, you can try to grep the HBase master log for "2077c5dfbfb97d67f09120e4b9cdc15a", looking for the last "OPEN" location. The HBase master UI might also tell you if this region is stuck in transition (RIT). If so, `hbase hbck` should be able to help you.
Created 08-08-2016 03:23 PM
How many nodes are in your cluster and how many are up? If nodes are up, then what about all HBase region server processes? Also follow Josh's suggestion and check the region server logs.
Created 08-23-2016 09:09 AM
Thanks Josh.