Hi Team,
We are receiving "java.io.FileNotFoundException: No such file or directory" exception and Region is in OFFLINE state. I tried to fix region using "hbck -fixAssignments <tablename>" and "hbck -repair". The region failed to up due to "Exception in thread "main" java.io.IOException: 1 region(s) could not be checked or repaired."
In logs, I found below error:
2019-05-21 13:46:47,548 ERROR [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000] master.MasterRpcServices: e.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2609)
... 9 more
Caused by: java.io.FileNotFoundException: No such file or directory 's3://bucketName/hbase-rootdir-v3/data/default/gdsVersion2_geospatialNodeEntity_id_v4/aa381866fb29f39ddf7d01ff029369af/d/478fb0436ad4403eb8cb8dd6aa006f73'
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:805)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.getFileStatus(EmrFileSystem.java:536)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:452)
at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:127)
at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:670)
at org.apache.hadoop.hbase.regionserver.HStore.commitFile(HStore.java:1016)
at org.apache.hadoop.hbase.regionserver.HStore.access$400(HStore.java:122)
at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2474)
... 10 more
The other continuous error which I am seeing is :
2019-05-21 13:47:07,679 ERROR [split-log-closeStream-3] wal.WALSplitter: Couldn't rename s3://bucketName/hbase-rootdir-v3/data/default/gdsVersion2_geospatialNodeEntity_attr_identifier_geom_timestamp_v8/82d7649c68f79c83dcb57253ebc5e3b7/recovered.edits/0000000000000088373-ip-10-1-8-232.ec2.internal%2C16020%2C1558413057371.1558446354676.temp to s3://bucketName/hbase-rootdir-v3/data/default/gdsVersion2_geospatialNodeEntity_attr_identifier_geom_timestamp_v8/82d7649c68f79c83dcb57253ebc5e3b7/recovered.edits/0000000000000088464
java.io.IOException: Cannot get log reader
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:365)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:277)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:265)
at org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.deleteOneWithFewerEntries(WALSplitter.java:1363)
at org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.closeWriter(WALSplitter.java:1496)
at org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink$2.call(WALSplitter.java:1448)
at org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink$2.call(WALSplitter.java:1445)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Wrong FS: s3://bucketName/hbase-rootdir-v3/data/default/gdsVersion2_geospatialNodeEntity_attr_identifier_geom_timestamp_v8/82d7649c68f79c83dcb57253ebc5e3b7/recovered.edits/0000000000000088464, expected: hdfs://ip-10-1-8-4.ec2.internal:8020
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:669)
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:214)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:337)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:790)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)
Team, Since I am new in Hbase and doing some prototype of our use-cases. Could you please help be to deepdive in the issue and repair the region.
