Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

mapred-default.xml and mapred-site.xml are not correctly merged in CDH 5.3.2

avatar
Expert Contributor

Hi, I'm running mapreduce job using hadoop jar command
The problem is that hadoop-core.jar contains mapred-default.xml with -Xmx200m for mapreduce.
I have correct client conf in /etc/hadoop/conf/mapred-site.xml
xmx is big enough there.
When job started,
mapred.child.java.opts -Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336
property is merged. -Xmx200m comes from bundled mapred-default.xml

and

-Djava.net.preferIPv4Stack=true -Xmx9448718336 comes from my config.

 

Job uses -Xmx200m for mappres and fails

What is the right way to exclude -Xmx200m and leave only -Xmx9448718336 from mapred-site.xml?

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Ha, looks like Camus runs local job runner, that is the problem...

Need to inform camus that we have yarn here.

View solution in original post

3 REPLIES 3

avatar
Mentor
The last specified -Xmx takes precedence, normally, in Oracle/Sun
JREs. What evidence in the failure logs is suggesting that the mapper
starts only with 200m heap instead?

avatar
Expert Contributor

Thanks for the reply.

There are several myctical problems:

1. Here is what ResourceManager conf says:

http://my.resource.manager.ru:8088/conf

<property>
<name>mapred.child.java.opts</name>
<value>-Xmx200m</value>
<source>mapred-default.xml</source>
</property>
 
I can't find any mapred-default.xml, only inside hadoop-core.jar which is in cloudera parcels
 
3. Here is running app:
Job configuration on NodeManerr UI says:
mapred.child.java.opts-Xmx200m -Djava.net.preferIPv4Stack=true -Xmx9448718336

 

but ps -ef | grep java says:

 

yarn     54070 53908 99 15:55 ?        00:08:20 /usr/java/jdk1.7.0_55/bin/java -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar camus-tool.jar com.linkedin.camus.etl.kafka.CamusJob -P camus.properties

 

NOw we get xmx as 1000m, which is still not enough, but we don't have such property...

 

 

 

avatar
Expert Contributor

Ha, looks like Camus runs local job runner, that is the problem...

Need to inform camus that we have yarn here.