Member since
09-20-2016
2
Posts
1
Kudos Received
0
Solutions
09-28-2016
07:05 AM
@Nitin Shelke thanks for the help , increasing value of "mapreduce.map.memory.mb to 2.75 GB from 2.5Gb worked but just for 400kb file whether 2.7GB really needed?. because of this current job the other parallel jobs getting delayed seems they are queued till current job to complete as this current job taking 2.75GB out of 5 GB available. is there any way to fine tune this scenario?
... View more
09-20-2016
05:34 AM
1 Kudo
In Hadoop v2,HDP2.3 stack, in single node cluster My machine has 2 cores and 14G memory out of which 5.6 G free . Now with YARN, when run the stream job application with 400kb input file on the same machine, I got container error.
Container [pid=58662,containerID=container_e03_1474273534378_0008_01_000002] is running beyond physical memory limits. Current usage: 2.6 GB of 2.5 GB physical memory used; 4.1 GB of 5.3 GB virtual memory used. Killing container. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 When i used the same script of input file 245 KB, it got succeeded. But failing for the the input file of size 400 kb and above. By default, I have this settings: yarn-site.xml -----------------
<property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>10240</value> </property>
<property> <name>yarn.scheduler.maximum-allocation-vcores</name>
<value>1</value> </property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2560</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>10240</value>
</property>
mapred-site.xml --------------------------- <property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx2048m</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2560</value>
</property> <property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx4096m</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>5120</value>
</property> kindly help on this.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache YARN
-
MapReduce