Support Questions
Find answers, ask questions, and share your expertise

HBase 1.2.0 ExportSnapshot failing - org.apache.hadoop.mapred.YarnChild: Error running child : java

New Contributor



  As I've been in trouble some time with this one, I share it, it may be useful to somebody else 🙂


  On CDH Quickstart 5.13, running HBase ExportSnapshot failed each time on this error :


18/09/06 06:40:21 INFO mapreduce.Job: Task Id : attempt_1536162453275_0013_m_000000_0, Status : FAILED
Error: Java heap space


  After three tries, the job stopped.




yarn logs -applicationId application_1536162453275_0013

helped me as the traces were more explicit :




2018-09-06 06:40:20,699 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2018-09-06 06:40:20,914 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportSnapshotInputFormat$ExportSnapshotInputSplit@44b35022
2018-09-06 06:40:21,042 INFO [main] org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper: Using bufferSize=128 M
2018-09-06 06:40:21,371 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyData(
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$
        at org.apache.hadoop.mapred.MapTask.runNewMapper(
        at org.apache.hadoop.mapred.YarnChild$
        at Method)
        at org.apache.hadoop.mapred.YarnChild.main(



  As explained in this post : looking at source code of the class in the corresponding version helped a lot but looking at the last version helped too, because in old code the crash happened when the buffersize were initialized and when no buffersize is provided the default is default block size (here we can see it : bufferSize=128 M).

  In the latest implementation, the bufferSize is a constant whose value is 64 k.


  Setting a value for the buffersize (65636) made the job successful

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -Dsnapshot.export.buffer.size=65536 -snapshot my_snapshot -copy-to hdfs://localhost/tmp/deploy_hbase08


Hope thie helps,