alternatives hadoop-conf - difference between Namenode and Datanodes

[#1]

Hi,


I've just realized there is a difference in configuration of alternatives functionality between Datanodes and the NameNode.

NameNode:

alternatives --display hadoop-conf                             
hadoop-conf - status is auto.
 link currently points to /etc/hadoop/conf.cloudera.hdfs                      
/etc/hadoop/conf.cloudera.hdfs - priority 90        
/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/etc/hadoop/conf.empty - priority 10                     
Current `best' version is /etc/hadoop/conf.cloudera.hdfs.

DataNode:
 alternatives --display hadoop-conf
hadoop-conf - status is auto.
 link currently points to /etc/hadoop/conf.cloudera.yarn
/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/etc/hadoop/conf.empty - priority 10
/etc/hadoop/conf.cloudera.yarn - priority 92
/etc/hadoop/conf.cloudera.hdfs - priority 90
Current `best' version is /etc/hadoop/conf.cloudera.yarn


On NameNode I see directory /etc/hadoop/conf.cloudera.yarn but it seems to be unused. For example, even though yarn.resourcemanager.address is set correctly in /etc/hadoop/conf.cloudera.yarn/yarn-site.xml I receive:

INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

when I try 'yarn application -list' from NameNode, and correct results when I try from DataNodes.
Is it expected behaviour? Has anybody hit this problem before?

[#2 · Accepted solution]

@szamasz,

Your observations are correct and expected.  The configuration differences are due to the logic used in Cloudera Manager's management of client configurations. Depending on what roles you have installed on each host, CM will determine the configuration files necessary for clients on those hosts.  If you have a YARN role (NodeManager or Resource Manager) running on the same host as the NameNode, you will probably see the same alternative link to conf.cloudera.yarn.

This is because the YARN configuration will contain client files for YARN as well as HDFS.

If you wish, you can go to your YARN service in Cloudera Manager and click on the Instances subtab.  Click Add new Role Instances and add a new role of "Gateway" to the NameNode host.  Then, deploy client configuration. You will then see that the alternatives link will change to /etc/hadoop/conf.cloudera.yarn

Adding the Gateway Role will allow clients (not Services) to read the current client configuration.

Servers do not use the client configuration files in /etc.  Rather, they use the configuration files deployed to the /var/run/cloudera-scm-agent/process directory when starting a service via Cloudera Manager.

I don't know if all of this will help you solve your problem since I didn't see what command you tried to run when the problem happened.  If it doesn't help, please describe what the problem is and we'll help you.

Ben

[#3]

Hi Ben,

Thank you very much for your explanations. Especially I appreciated your clarification about service/client configuration difference and how to enable client configuration for yarn. It solves my issue, thanks a million.

Regards,

Sz