HBase Export Snapshot to S3A Fails Due to Memory Leaks After Upgrade to CM 5.14.0

[#1]

Hello,

We upgraded from CM 5.7.6 to CM 5.14.0 and we can no longer run HBase Export Snapshot to S3 bucket via s3a:// file system because it always fails in the VerifySnapshot step with OutOfMemoryError and a heap dump.

After tracing changes in source code we might have found HBase/Hadoop AWS issue related to metric reporting in S3AInstrumentation. More information could be found here:

• https://issues.apache.org/jira/browse/HADOOP-15392
• https://issues.apache.org/jira/browse/HBASE-20433

As a work-around we increased client heap size extensively, but it's not a scalable solution. We were able to run backups with default 256 MiB heap size in CM 5.7.6.

Is there any option we can adjust in order to achieve permanent solution?

[#2]

Thanks for filing a bug on this. We are looking at it and will let you know when we have a fix or workaround.