Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎07-06-2018

HBase 1.2.0 ExportSnapshot failing - org.apache.hadoop.mapred.YarnChild: Error running child : java

[ Edited ]

Hi,

 

  As I've been in trouble some time with this one, I share it, it may be useful to somebody else :)

 

  On CDH Quickstart 5.13, running HBase ExportSnapshot failed each time on this error :

 

18/09/06 06:40:21 INFO mapreduce.Job: Task Id : attempt_1536162453275_0013_m_000000_0, Status : FAILED
Error: Java heap space

 

  After three tries, the job stopped.

 

using

 

yarn logs -applicationId application_1536162453275_0013

helped me as the traces were more explicit :

 

 

 

2018-09-06 06:40:20,699 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2018-09-06 06:40:20,914 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportSnapshotInputFormat$ExportSnapshotInputSplit@44b35022
2018-09-06 06:40:21,042 INFO [main] org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper: Using bufferSize=128 M
2018-09-06 06:40:21,371 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyData(ExportSnapshot.java:381)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:284)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:212)
        at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:130)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)


 

 

  As explained in this post : https://community.cloudera.com/t5/Storage-Random-Access-HDFS/HBase-exportSnapshot-failing-need-help-... looking at source code of the class in the corresponding version helped a lot but looking at the last version helped too, because in old code the crash happened when the buffersize were initialized and when no buffersize is provided the default is default block size (here we can see it : bufferSize=128 M).

  In the latest implementation, the bufferSize is a constant whose value is 64 k.

 

  Setting a value for the buffersize (65636) made the job successful

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -Dsnapshot.export.buffer.size=65536 -snapshot my_snapshot -copy-to hdfs://localhost/tmp/deploy_hbase08

 

Hope thie helps,

 

Regards,

 

Christophe

Announcements