In Hadoop v2,HDP2.3 stack, in single node cluster My machine has 2 cores and 14G memory out of which 5.6 G free . Now with YARN, when run the stream job application with 400kb input file on the same machine, I got container error.
Container [pid=58662,containerID=container_e03_1474273534378_0008_01_000002] is running beyond physical memory limits. Current usage: 2.6 GB of 2.5 GB physical memory used; 4.1 GB of 5.3 GB virtual memory used. Killing container. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143
When i used the same script of input file 245 KB, it got succeeded. But failing for the the input file of size 400 kb and above.
By default, I have this settings:
yarn-site.xml
-----------------
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>10240</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>1</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2560</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>10240</value>
</property>
mapred-site.xml
---------------------------
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx2048m</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2560</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx4096m</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>5120</value>
</property>
kindly help on this.
