Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎01-10-2019

Hbase rename error sometimes when connecting to s3a

We are using a s3 storage for backend data store.

When running hbase for testing ,it will throws rename error and crashes sometimes. 

2683120.scsp01881%2C16020%2C1548772683120.meta.regiongroup-0.1548773314620 to s3a://hbase/oldWALs/scsp01881%2C16020%2C1548772683120.scsp01881%2C16020%2C1548772683120.me

ta.regiongroup-0.1548773314620

2019-01-29 23:54:31,646 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ***** ABORTING region server scsp01881,16020,1548772683120: IOE in log roller *****

java.io.IOException: Unable to rename s3a://hbase/WALs/scsp01881,16020,1548772683120/scsp01881%2C16020%2C1548772683120.scsp01881%2C16020%2C1548772683120.meta.regiongrou

p-0.1548773314620 to s3a://hbase/oldWALs/scsp01881%2C16020%2C1548772683120.scsp01881%2C16020%2C1548772683120.meta.regiongroup-0.1548773314620

       at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archiveLogFile(AbstractFSWAL.java:651)

       at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.cleanOldLogs(AbstractFSWAL.java:627)

       at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:767)

       at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:184)

       at java.lang.Thread.run(Thread.java:748)

 

There is a bug reported on Hbase Jira site and it seems to be fixed in Hadoop 3.1.

https://issues.apache.org/jira/browse/HADOOP-14161

 

But the CDH 6.1.0 comes with Hadoop 3.0. Is there any workaround for this issue?