We upgraded from CM 5.7.6 to CM 5.14.0 and we can no longer run HBase Export Snapshot to S3 bucket via s3a:// file system because it always fails in the VerifySnapshot step with OutOfMemoryError and a heap dump.
After tracing changes in source code we might have found HBase/Hadoop AWS issue related to metric reporting in S3AInstrumentation. More information could be found here:
As a work-around we increased client heap size extensively, but it's not a scalable solution. We were able to run backups with default 256 MiB heap size in CM 5.7.6.
Is there any option we can adjust in order to achieve permanent solution?
Thanks for filing a bug on this. We are looking at it and will let you know when we have a fix or workaround.