Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase rename error sometimes when connecting to s3a

Highlighted

Hbase rename error sometimes when connecting to s3a

New Contributor

We are using a s3 storage for backend data store.

When running hbase for testing ,it will throws rename error and crashes sometimes. 

2683120.scsp01881%2C16020%2C1548772683120.meta.regiongroup-0.1548773314620 to s3a://hbase/oldWALs/scsp01881%2C16020%2C1548772683120.scsp01881%2C16020%2C1548772683120.me

ta.regiongroup-0.1548773314620

2019-01-29 23:54:31,646 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ***** ABORTING region server scsp01881,16020,1548772683120: IOE in log roller *****

java.io.IOException: Unable to rename s3a://hbase/WALs/scsp01881,16020,1548772683120/scsp01881%2C16020%2C1548772683120.scsp01881%2C16020%2C1548772683120.meta.regiongrou

p-0.1548773314620 to s3a://hbase/oldWALs/scsp01881%2C16020%2C1548772683120.scsp01881%2C16020%2C1548772683120.meta.regiongroup-0.1548773314620

       at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archiveLogFile(AbstractFSWAL.java:651)

       at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.cleanOldLogs(AbstractFSWAL.java:627)

       at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:767)

       at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:184)

       at java.lang.Thread.run(Thread.java:748)

 

There is a bug reported on Hbase Jira site and it seems to be fixed in Hadoop 3.1.

https://issues.apache.org/jira/browse/HADOOP-14161

 

But the CDH 6.1.0 comes with Hadoop 3.0. Is there any workaround for this issue?