Reply
Expert Contributor
Posts: 162
Registered: ‎07-29-2013
Accepted Solution

mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Hi, I'm running mapreduce job using hadoop jar command
The problem is that hadoop-core.jar contains mapred-default.xml with -Xmx200m for mapreduce.
I have correct client conf in /etc/hadoop/conf/mapred-site.xml
xmx is big enough there.
When job started,
mapred.child.java.opts -Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336
property is merged. -Xmx200m comes from bundled mapred-default.xml

and

-Djava.net.preferIPv4Stack=true -Xmx9448718336 comes from my config.

 

Job uses -Xmx200m for mappres and fails

What is the right way to exclude -Xmx200m and leave only -Xmx9448718336 from mapred-site.xml?

Posts: 1,892
Kudos: 432
Solutions: 302
Registered: ‎07-31-2013

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

The last specified -Xmx takes precedence, normally, in Oracle/Sun
JREs. What evidence in the failure logs is suggesting that the mapper
starts only with 200m heap instead?

Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Thanks for the reply.

There are several myctical problems:

1. Here is what ResourceManager conf says:

http://my.resource.manager.ru:8088/conf

<property>
<name>mapred.child.java.opts</name>
<value>-Xmx200m</value>
<source>mapred-default.xml</source>
</property>
 
I can't find any mapred-default.xml, only inside hadoop-core.jar which is in cloudera parcels
 
3. Here is running app:
Job configuration on NodeManerr UI says:
mapred.child.java.opts-Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336

 

but ps -ef | grep java says:

 

yarn     54070 53908 99 15:55 ?        00:08:20 /usr/java/jdk1.7.0_55/bin/java -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar camus-tool.jar com.linkedin.camus.etl.kafka.CamusJob -P camus.properties

 

NOw we get xmx as 1000m, which is still not enough, but we don't have such property...

 

 

 
Highlighted
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Ha, looks like Camus runs local job runner, that is the problem...

Need to inform camus that we have yarn here.

Announcements