Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Region went down due to No such file or directory S3 Exception.

Highlighted

Region went down due to No such file or directory S3 Exception.

New Contributor

Hi Team,


We are receiving "java.io.FileNotFoundException: No such file or directory" exception and Region is in OFFLINE state. I tried to fix region using "hbck -fixAssignments <tablename>" and "hbck -repair". The region failed to up due to "Exception in thread "main" java.io.IOException: 1 region(s) could not be checked or repaired."


In logs, I found below error:

2019-05-21 13:46:47,548 ERROR [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000] master.MasterRpcServices: e.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2609)
    ... 9 more
Caused by: java.io.FileNotFoundException: No such file or directory 's3://bucketName/hbase-rootdir-v3/data/default/gdsVersion2_geospatialNodeEntity_id_v4/aa381866fb29f39ddf7d01ff029369af/d/478fb0436ad4403eb8cb8dd6aa006f73'
    at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:805)
    at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.getFileStatus(EmrFileSystem.java:536)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:452)
    at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:127)
    at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:670)
    at org.apache.hadoop.hbase.regionserver.HStore.commitFile(HStore.java:1016)
    at org.apache.hadoop.hbase.regionserver.HStore.access$400(HStore.java:122)
    at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2474)
    ... 10 more


The other continuous error which I am seeing is :


2019-05-21 13:47:07,679 ERROR [split-log-closeStream-3] wal.WALSplitter: Couldn't rename s3://bucketName/hbase-rootdir-v3/data/default/gdsVersion2_geospatialNodeEntity_attr_identifier_geom_timestamp_v8/82d7649c68f79c83dcb57253ebc5e3b7/recovered.edits/0000000000000088373-ip-10-1-8-232.ec2.internal%2C16020%2C1558413057371.1558446354676.temp to s3://bucketName/hbase-rootdir-v3/data/default/gdsVersion2_geospatialNodeEntity_attr_identifier_geom_timestamp_v8/82d7649c68f79c83dcb57253ebc5e3b7/recovered.edits/0000000000000088464
java.io.IOException: Cannot get log reader
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:365)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:277)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:265)
    at org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.deleteOneWithFewerEntries(WALSplitter.java:1363)
    at org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.closeWriter(WALSplitter.java:1496)
    at org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink$2.call(WALSplitter.java:1448)
    at org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink$2.call(WALSplitter.java:1445)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Wrong FS: s3://bucketName/hbase-rootdir-v3/data/default/gdsVersion2_geospatialNodeEntity_attr_identifier_geom_timestamp_v8/82d7649c68f79c83dcb57253ebc5e3b7/recovered.edits/0000000000000088464, expected: hdfs://ip-10-1-8-4.ec2.internal:8020
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:669)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:214)
    at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)
    at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:337)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:790)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:303)


Team, Since I am new in Hbase and doing some prototype of our use-cases. Could you please help be to deepdive in the issue and repair the region.