We have an application managed by yarn when we change yarn-site.xml those changes are not reflected , application is still running with old configuration. We are new to Yarn any help in this regard will be helpful
Note : we have already tried restarted yarn using stop-yarn.sh and start-yarn.sh also restared dfs using start-dfs.sh and stop-dfs.sh . We are using hadoop 2.7.3
this is what yarn looks with only max memory configured to 16GB as shown in the picture but actual configuration is 22GB as per yarn-site.xml
this is the yarn-site.xml
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>hdfs-name-node</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>21528</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>6</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>21528</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>file:///tmp/hadoop/data/nm-local-dir,file:///tmp/hadoop/data/nm-local-dir/filecache,file:///tmp/hadoop/data/nm-local-dir/usercache</value> </property> <property> <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name> <value>500</value> </property> <property> <name>yarn.nodemanager.localizer.cache.target-size-mb</name> <value>512</value> </property> </configuration>
this is the node configuration
1 Master/Driver Node : Memory :24GB Cores :8 4 Worker Nodes : Memory :24GB Cores :8
Are you managing hadoop services through ambari or it's manual installation?
if it's ambari managed cluster, you have to make configuration changes through ambari only, otherwise your changes will overwrite by Ambari
Can you give a little background of your Cluster or node ? From your input it seems you are an Ambari or Cloudera Manager managed hadoop installation. application managed by YARN ? when you start with shell scripts that it is a manually installed Hadoop otherwise you should do all changes and stop/start through Ambari or cloudera Manager.
With more details the community will defintitely come to your rescue !
@Geoffrey Shelton Okot we are not using Ambari or Cloudera Manager , we are using apache hadoop 2.7.3 all changes i'm making are through cli , i have added additional info to the question please let me know if any more info is required
You can check if the values you configured are actually in use with this command:
hdfs getconf -confKey yarn.nodemanager.resource.memory-mb
or from the ResourceManager UI: <rm.host>:8088/conf
If the values are in sync, you can also check the ResourceManager logs for further information.