Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Solved Go to solution

mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Expert Contributor

Hi, I'm running mapreduce job using hadoop jar command
The problem is that hadoop-core.jar contains mapred-default.xml with -Xmx200m for mapreduce.
I have correct client conf in /etc/hadoop/conf/mapred-site.xml
xmx is big enough there.
When job started,
mapred.child.java.opts -Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336
property is merged. -Xmx200m comes from bundled mapred-default.xml

and

-Djava.net.preferIPv4Stack=true -Xmx9448718336 comes from my config.

 

Job uses -Xmx200m for mappres and fails

What is the right way to exclude -Xmx200m and leave only -Xmx9448718336 from mapred-site.xml?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Expert Contributor

Ha, looks like Camus runs local job runner, that is the problem...

Need to inform camus that we have yarn here.

3 REPLIES 3

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Master Guru
The last specified -Xmx takes precedence, normally, in Oracle/Sun
JREs. What evidence in the failure logs is suggesting that the mapper
starts only with 200m heap instead?

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Expert Contributor

Thanks for the reply.

There are several myctical problems:

1. Here is what ResourceManager conf says:

http://my.resource.manager.ru:8088/conf

<property>
<name>mapred.child.java.opts</name>
<value>-Xmx200m</value>
<source>mapred-default.xml</source>
</property>
 
I can't find any mapred-default.xml, only inside hadoop-core.jar which is in cloudera parcels
 
3. Here is running app:
Job configuration on NodeManerr UI says:
mapred.child.java.opts-Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336

 

but ps -ef | grep java says:

 

yarn     54070 53908 99 15:55 ?        00:08:20 /usr/java/jdk1.7.0_55/bin/java -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar camus-tool.jar com.linkedin.camus.etl.kafka.CamusJob -P camus.properties

 

NOw we get xmx as 1000m, which is still not enough, but we don't have such property...

 

 

 

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Expert Contributor

Ha, looks like Camus runs local job runner, that is the problem...

Need to inform camus that we have yarn here.