question Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2 in Archives of Support Questions (Read Only)

mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

sergey.sheypak566881637 — Fri, 16 Sep 2022 09:28:01 GMT

Hi, I'm running mapreduce job using hadoop jar command
The problem is that hadoop-core.jar contains mapred-default.xml with -Xmx200m for mapreduce.
I have correct client conf in /etc/hadoop/conf/mapred-site.xml
xmx is big enough there.
When job started,
mapred.child.java.opts -Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336
property is merged. -Xmx200m comes from bundled mapred-default.xml

and

-Djava.net.preferIPv4Stack=true -Xmx9448718336 comes from my config.

Job uses -Xmx200m for mappres and fails

What is the right way to exclude -Xmx200m and leave only -Xmx9448718336 from mapred-site.xml?

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Harsh J — Wed, 06 May 2015 11:11:11 GMT

The last specified -Xmx takes precedence, normally, in Oracle/Sun
JREs. What evidence in the failure logs is suggesting that the mapper
starts only with 200m heap instead?

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

sergey.sheypak566881637 — Wed, 06 May 2015 12:59:55 GMT

Thanks for the reply.

There are several myctical problems:

1. Here is what ResourceManager conf says:

http://my.resource.manager.ru:8088/conf

<name>mapred.child.java.opts</name>
<value>-Xmx200m</value>
<source>mapred-default.xml</source>

</property>

I can't find any mapred-default.xml, only inside hadoop-core.jar which is in cloudera parcels

3. Here is running app:

Job configuration on NodeManerr UI says:

mapred.child.java.opts

-Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336

but ps -ef | grep java says:

yarn 54070 53908 99 15:55 ? 00:08:20 /usr/java/jdk1.7.0_55/bin/java -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar camus-tool.jar com.linkedin.camus.etl.kafka.CamusJob -P camus.properties

NOw we get xmx as 1000m, which is still not enough, but we don't have such property...

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

sergey.sheypak566881637 — Wed, 06 May 2015 13:52:17 GMT

Ha, looks like Camus runs local job runner, that is the problem...

Need to inform camus that we have yarn here.