Support Questions

Find answers, ask questions, and share your expertise
Announcements
Welcome to the upgraded Community! Read this blog to see What’s New!

Yarn-site.xml changes not reflecting

avatar
Explorer

We have an application managed by yarn when we change yarn-site.xml those changes are not reflected , application is still running with old configuration. We are new to Yarn any help in this regard will be helpful

Note : we have already tried restarted yarn using stop-yarn.sh and start-yarn.sh also restared dfs using start-dfs.sh and stop-dfs.sh . We are using hadoop 2.7.3

this is what yarn looks with only max memory configured to 16GB as shown in the picture but actual configuration is 22GB as per yarn-site.xml

12037-yarn-executorcount.png

this is the yarn-site.xml

<configuration>
    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hdfs-name-node</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>21528</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>6</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>21528</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>file:///tmp/hadoop/data/nm-local-dir,file:///tmp/hadoop/data/nm-local-dir/filecache,file:///tmp/hadoop/data/nm-local-dir/usercache</value>
    </property>
    <property>
        <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
        <value>500</value>
    </property>
    <property>
         <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
         <value>512</value>
    </property>
    </configuration>

this is the node configuration

1 Master/Driver Node : 
  Memory :24GB Cores :8 

4 Worker Nodes : 
  Memory :24GB Cores :8

7 REPLIES 7

avatar

@Aditya Mamidala

Are you managing hadoop services through ambari or it's manual installation?

if it's ambari managed cluster, you have to make configuration changes through ambari only, otherwise your changes will overwrite by Ambari

avatar
Explorer

@Divakar Annapureddy We are not using amabri for management ...we are using apache hadoop 2.7.3

avatar
@Aditya Mamidala

Try to make configuration changes with Yarn user?

use below commands to login as yarn:

su - root and then

su - yarn

avatar
Mentor

@Aditya Mamidala

Can you give a little background of your Cluster or node ? From your input it seems you are an Ambari or Cloudera Manager managed hadoop installation. application managed by YARN ? when you start with shell scripts that it is a manually installed Hadoop otherwise you should do all changes and stop/start through Ambari or cloudera Manager.

With more details the community will defintitely come to your rescue !

avatar
Explorer

@Geoffrey Shelton Okot we are not using Ambari or Cloudera Manager , we are using apache hadoop 2.7.3 all changes i'm making are through cli , i have added additional info to the question please let me know if any more info is required

avatar
Mentor

@Aditya Mamidala

Here is a good reference for manually setting the YARN memory parameters

avatar
Expert Contributor

You can check if the values you configured are actually in use with this command:

hdfs getconf -confKey yarn.nodemanager.resource.memory-mb

or from the ResourceManager UI: <rm.host>:8088/conf

If the values are in sync, you can also check the ResourceManager logs for further information.

Labels