Reply
Expert Contributor
Posts: 162
Registered: ‎07-29-2013
Accepted Solution

mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Hi, I'm running mapreduce job using hadoop jar command
The problem is that hadoop-core.jar contains mapred-default.xml with -Xmx200m for mapreduce.
I have correct client conf in /etc/hadoop/conf/mapred-site.xml
xmx is big enough there.
When job started,
mapred.child.java.opts -Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336
property is merged. -Xmx200m comes from bundled mapred-default.xml

and

-Djava.net.preferIPv4Stack=true -Xmx9448718336 comes from my config.

 

Job uses -Xmx200m for mappres and fails

What is the right way to exclude -Xmx200m and leave only -Xmx9448718336 from mapred-site.xml?

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

The last specified -Xmx takes precedence, normally, in Oracle/Sun
JREs. What evidence in the failure logs is suggesting that the mapper
starts only with 200m heap instead?

Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Thanks for the reply.

There are several myctical problems:

1. Here is what ResourceManager conf says:

http://my.resource.manager.ru:8088/conf

<property>
<name>mapred.child.java.opts</name>
<value>-Xmx200m</value>
<source>mapred-default.xml</source>
</property>
 
I can't find any mapred-default.xml, only inside hadoop-core.jar which is in cloudera parcels
 
3. Here is running app:
Job configuration on NodeManerr UI says:
mapred.child.java.opts-Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336

 

but ps -ef | grep java says:

 

yarn     54070 53908 99 15:55 ?        00:08:20 /usr/java/jdk1.7.0_55/bin/java -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar camus-tool.jar com.linkedin.camus.etl.kafka.CamusJob -P camus.properties

 

NOw we get xmx as 1000m, which is still not enough, but we don't have such property...

 

 

 
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

Ha, looks like Camus runs local job runner, that is the problem...

Need to inform camus that we have yarn here.

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.