Welcome to the Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

Unable to change the container memory settings in cloudera manager installation

avatar
Explorer

I have a 3 node cluster set up using the Cloudera Manager Installer Cloudera 5.5.x in Cloudera Express mode.

I have changed the default values and set the values below for the following variables in the configuration screen for YARN.

mapreduce.map.java.opts -Djava.net.preferIPv4Stack=true -Xmx3276m
mapreduce.reduce.java.opts -Djava.net.preferIPv4Stack=true -Xmx3276m
mapreduce.map.memory.mb   4GiB
mapreduce.reduce.memory.mb 4GiB
mapreduce.map.java.opts.max.heap 3276 MiB
mapreduce.reduce.java.opts.max.heap 3276 MiB
yarn.scheduler.maximum-allocation-mb 4GiB

I am trying to set the max container memory to 4GB and the max map|reduce heap size to 3276MB as can be seen in the values above, and after setting these values I have also restarted the cluster, however the mapreduce jobs fail with the following error message as the max container memory is still set to 1GB, can someone please let me know if this a known issue with CM/CDH Cloudera 5.5.x when installed in Cloudera Express mode?

15/12/29 06:06:10 INFO mapreduce.Job: Task Id : attempt_1451388750423_0009_m_000003_1, Status : FAILED
Container [pid=31644,containerID=container_1451388750423_0009_01_000022] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.5 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1451388750423_0009_01_000022 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 31649 31644 31644 31644 (java) 54081 688 1530179584 262135 /usr/java/jdk1.7.0_67-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true -Xmx820m -Djava.io.tmpdir=/yarn/nm/usercache/gd/appcache/application_1451388750423_0009/container_1451388750423_0009_01_000022/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1451388750423_0009/container_1451388750423_0009_01_000022 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 69.30.221.130 55260 attempt_1451388750423_0009_m_000003_1 22 
    |- 31644 31643 31644 31644 (bash) 0 0 108650496 297 /bin/bash -c /usr/java/jdk1.7.0_67-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Xmx820m -Djava.io.tmpdir=/yarn/nm/usercache/gd/appcache/application_1451388750423_0009/container_1451388750423_0009_01_000022/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1451388750423_0009/container_1451388750423_0009_01_000022 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 69.30.221.130 55260 attempt_1451388750423_0009_m_000003_1 22 1>/var/log/hadoop-yarn/container/application_1451388750423_0009/container_1451388750423_0009_01_000022/stdout 2>/var/log/hadoop-yarn/container/application_1451388750423_0009/container_1451388750423_0009_01_000022/stderr 
Who agreed with this topic