Reply
Highlighted
Explorer
Posts: 22
Registered: ‎04-12-2016

Reducer failed on java heap size

Hi all,

I have MapRed job that compacted small processed files in the HDFS.

In the Reducer log i see the following error and below is the workflow.xml parameters.

I tried to fine tune the paarameters of the memory but it's not helping.

Can you please help to understand what i'm missing?

=========================================================================

2016-08-15 17:43:48,981 INFO org.apache.hadoop.mapred.Merger: Merging 707 sorted segments
2016-08-15 17:43:49,522 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.lzo_deflate]
2016-08-15 17:43:49,531 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.lzo_deflate]
2016-08-15 17:43:49,554 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.lzo_deflate]
2016-08-15 17:43:49,606 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.lzo_deflate]
2016-08-15 17:43:49,628 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.lzo_deflate
2016-08-15 17:43:53,337 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.lzo_deflate]
2016-08-15 17:43:53,352 INFO org.apache.hadoop.mapred.Merger: Merging 234 intermediate segments out of a total of 486
2016-08-15 17:59:43,887 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2016-08-15 17:59:43,889 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:357)
at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:419)
at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:238)
at org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:348)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:368)
at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:517)
at org.apache.hadoop.mapred.Merger.merge(Merger.java:142)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.createKVIterator(ReduceTask.java:2537)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:425)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)

I have the following parameters in the following in the workflow:

arg>-Dmapred.reduce.child.java.opts=-Xmx6144M</arg>
<arg>-Dmapred.map.child.java.opts=-Xmx4096M</arg>
<arg>-Dmapred.child.java.opts=-Xmx6144M</arg>
<arg>-Dmapred.job.shuffle.input.buffer.percent=0.30</arg>
<arg>-Dio.sort.factor=12</arg>
<arg>-Dio.sort.mb=256</arg>

Explorer
Posts: 22
Registered: ‎04-12-2016

Re: Reducer failed on java heap size

Who can help here please