- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Yarn-site.xml changes not reflecting
- Labels:
-
Apache Hadoop
-
Apache YARN
Created on 02-02-2017 01:17 PM - edited 08-18-2019 05:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have an application managed by yarn when we change yarn-site.xml those changes are not reflected , application is still running with old configuration. We are new to Yarn any help in this regard will be helpful
Note : we have already tried restarted yarn using stop-yarn.sh and start-yarn.sh also restared dfs using start-dfs.sh and stop-dfs.sh . We are using hadoop 2.7.3
this is what yarn looks with only max memory configured to 16GB as shown in the picture but actual configuration is 22GB as per yarn-site.xml
this is the yarn-site.xml
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>hdfs-name-node</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>21528</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>6</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>21528</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>file:///tmp/hadoop/data/nm-local-dir,file:///tmp/hadoop/data/nm-local-dir/filecache,file:///tmp/hadoop/data/nm-local-dir/usercache</value> </property> <property> <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name> <value>500</value> </property> <property> <name>yarn.nodemanager.localizer.cache.target-size-mb</name> <value>512</value> </property> </configuration>
this is the node configuration
1 Master/Driver Node : Memory :24GB Cores :8 4 Worker Nodes : Memory :24GB Cores :8
Created 02-02-2017 02:10 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Are you managing hadoop services through ambari or it's manual installation?
if it's ambari managed cluster, you have to make configuration changes through ambari only, otherwise your changes will overwrite by Ambari
Created 02-02-2017 04:56 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Divakar Annapureddy We are not using amabri for management ...we are using apache hadoop 2.7.3
Created 02-02-2017 05:20 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Try to make configuration changes with Yarn user?
use below commands to login as yarn:
su - root and then
su - yarn
Created 02-02-2017 05:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you give a little background of your Cluster or node ? From your input it seems you are an Ambari or Cloudera Manager managed hadoop installation. application managed by YARN ? when you start with shell scripts that it is a manually installed Hadoop otherwise you should do all changes and stop/start through Ambari or cloudera Manager.
With more details the community will defintitely come to your rescue !
Created 02-02-2017 07:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Geoffrey Shelton Okot we are not using Ambari or Cloudera Manager , we are using apache hadoop 2.7.3 all changes i'm making are through cli , i have added additional info to the question please let me know if any more info is required
Created 02-04-2017 09:36 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here is a good reference for manually setting the YARN memory parameters
Created 02-06-2017 02:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can check if the values you configured are actually in use with this command:
hdfs getconf -confKey yarn.nodemanager.resource.memory-mb
or from the ResourceManager UI: <rm.host>:8088/conf
If the values are in sync, you can also check the ResourceManager logs for further information.