Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to change the container memory settings in cloudera manager installation

Unable to change the container memory settings in cloudera manager installation

Explorer

I have a 3 node cluster set up using the Cloudera Manager Installer Cloudera 5.5.x in Cloudera Express mode.

I have changed the default values and set the values below for the following variables in the configuration screen for YARN.

mapreduce.map.java.opts -Djava.net.preferIPv4Stack=true -Xmx3276m
mapreduce.reduce.java.opts -Djava.net.preferIPv4Stack=true -Xmx3276m
mapreduce.map.memory.mb   4GiB
mapreduce.reduce.memory.mb 4GiB
mapreduce.map.java.opts.max.heap 3276 MiB
mapreduce.reduce.java.opts.max.heap 3276 MiB
yarn.scheduler.maximum-allocation-mb 4GiB

I am trying to set the max container memory to 4GB and the max map|reduce heap size to 3276MB as can be seen in the values above, and after setting these values I have also restarted the cluster, however the mapreduce jobs fail with the following error message as the max container memory is still set to 1GB, can someone please let me know if this a known issue with CM/CDH Cloudera 5.5.x when installed in Cloudera Express mode?

15/12/29 06:06:10 INFO mapreduce.Job: Task Id : attempt_1451388750423_0009_m_000003_1, Status : FAILED
Container [pid=31644,containerID=container_1451388750423_0009_01_000022] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.5 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1451388750423_0009_01_000022 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 31649 31644 31644 31644 (java) 54081 688 1530179584 262135 /usr/java/jdk1.7.0_67-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true -Xmx820m -Djava.io.tmpdir=/yarn/nm/usercache/gd/appcache/application_1451388750423_0009/container_1451388750423_0009_01_000022/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1451388750423_0009/container_1451388750423_0009_01_000022 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 69.30.221.130 55260 attempt_1451388750423_0009_m_000003_1 22 
    |- 31644 31643 31644 31644 (bash) 0 0 108650496 297 /bin/bash -c /usr/java/jdk1.7.0_67-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Xmx820m -Djava.io.tmpdir=/yarn/nm/usercache/gd/appcache/application_1451388750423_0009/container_1451388750423_0009_01_000022/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1451388750423_0009/container_1451388750423_0009_01_000022 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 69.30.221.130 55260 attempt_1451388750423_0009_m_000003_1 22 1>/var/log/hadoop-yarn/container/application_1451388750423_0009/container_1451388750423_0009_01_000022/stdout 2>/var/log/hadoop-yarn/container/application_1451388750423_0009/container_1451388750423_0009_01_000022/stderr 
2 REPLIES 2

Re: Unable to change the container memory settings in cloudera manager installation

Cloudera Employee

Have you also deployed the client configurations?  Is the node you are submitting the job from configured with a YARN gateway role?

 

The MapReduce configurations for YARN are gateway configurations, so restarting the cluster on its own is not enough to reflect the changes you made.  Check the /etc/hadoop/conf directory on the node you are submitting the jobs from to verify.

Re: Unable to change the container memory settings in cloudera manager installation

Explorer

 

Thanks for the reply , but I am sorry it is not very clear to me , can you please provide any link to the required steps , in this case.

 

Yoru help is appreciated!

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here