question unable to start nodemanager in CDH 5.16.2 in Support Questions

question unable to start nodemanager in CDH 5.16.2 in Support Questions https://community.cloudera.com/t5/Support-Questions/unable-to-start-nodemanager-in-CDH-5-16-2/m-p/299957#M219952 Hi All,   I am trying to add new nodes to my existing cluster ( which was built by Cloudera Director ) , new node was getting added with the help of Cloudera Manager manually.   When I try to add new node the NodeManager is not getting started but Cloudera Manager agent thinks that it started and running , but it's neither getting connected to resource manager nor even writing any logs ( /var/log/hadoop-yarn/ ) to log directory making it difficult to find what was going on.   NodeManager runs successfully in the nodes which are built by Cloudera Director but it's not running in the nodes added manually with the help of Cloudera Manager.   I could only see `stderr.log` and no other logs were found. I compared this file with the node running successfully and it all looks good.   Since the node is not producing any logs, I couldn't provide any logs from /var/log/hadoop-yarn/*NODEMANAGER*log.out file.   But following are the logs from cloudera-scm-agent.log ( which doesn't show any error , except after some time throws an connection refused on 8042 since NodeManager was not running)       <LI-CODE lang="markup">[18/Jul/2020 16:57:40 +0000] 10596 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.02 max:0.03 LIFE_MAX:0.06 [18/Jul/2020 17:05:03 +0000] 10596 CP Server Thread-6 _cplogging INFO 10.2.6.75 - - [18/Jul/2020:17:05:03] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0" [18/Jul/2020 17:05:03 +0000] 10596 MainThread process INFO [135-yarn-NODEMANAGER] Updating process: False {u'running': (True, False), u'run_generation ': (1, 2)} [18/Jul/2020 17:05:03 +0000] 10596 MainThread process INFO Deactivating process 135-yarn-NODEMANAGER [18/Jul/2020 17:05:04 +0000] 10596 MainThread agent INFO Triggering supervisord update. [18/Jul/2020 17:05:05 +0000] 10596 CP Server Thread-7 _cplogging INFO 10.2.6.75 - - [18/Jul/2020:17:05:05] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0" [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO [135-yarn-NODEMANAGER] Updating process. [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO Deleting process 135-yarn-NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO Triggering supervisord update. [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO [135-yarn-NODEMANAGER] Orphaning process [18/Jul/2020 17:05:05 +0000] 10596 MainThread util INFO Using generic audit plugin for process yarn-NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread util INFO Creating metadata plugin for process yarn-NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread util INFO Using specific metadata plugin for process yarn-NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread util INFO Using generic metadata plugin for process yarn-NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread util INFO Creating profile plugin for process yarn-NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread util INFO Using generic profile plugin for process yarn-NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO [136-yarn-NODEMANAGER] Instantiating process [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO [136-yarn-NODEMANAGER] Updating process: True {} [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO First time to activate the process [136-yarn-NODEMANAGER]. [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO Created /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO Chowning /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER to yarn (983) hadoop (990) [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO Chmod'ing /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER to 0751 [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO Created /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER/logs [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO Chowning /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER/logs to yarn (983) hadoop (990) [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO Chmod'ing /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER/logs to 0751 [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO [136-yarn-NODEMANAGER] Refreshing process files: None [18/Jul/2020 17:05:05 +0000] 10596 MainThread parcel INFO prepare_environment begin: {u'SPARK2': u'2.4.0.cloudera2-1.cdh5.13.3.p0.1041012', u'CDH': u'5.16.2-1.cdh5.16.2.p0.8'}, [u'cdh'], [u'hdfs-client-plugin', u'cdh-plugin', u'yarn-plugin', u'mapreduce2-plugin'] [18/Jul/2020 17:05:05 +0000] 10596 MainThread parcel INFO The following requested parcels are not available: {} [18/Jul/2020 17:05:05 +0000] 10596 MainThread parcel INFO Obtained tags ['spark2'] for parcel SPARK2 [18/Jul/2020 17:05:05 +0000] 10596 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'kudu', 'sentry', 'solr', 'spark'] for parcel CDH [18/Jul/2020 17:05:05 +0000] 10596 MainThread parcel INFO prepare_environment end: {'CDH': '5.16.2-1.cdh5.16.2.p0.8'} [18/Jul/2020 17:05:05 +0000] 10596 MainThread __init__ INFO Extracted 18 files and 0 dirs to /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER. [18/Jul/2020 17:05:05 +0000] 10596 MainThread __init__ INFO Copied special file /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER/topology.map to /etc/hadoop/conf.cloudera.CD-YARN-PbpthEaX/topology.map. [18/Jul/2020 17:05:05 +0000] 10596 MainThread __init__ INFO Copied special file /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER/topology.py to /etc/hadoop/conf.cloudera.CD-YARN-PbpthEaX/topology.py. [18/Jul/2020 17:05:05 +0000] 10596 MainThread __init__ INFO Copied special file /run/cloudera-scm-agent/process/136-yarn-NODEMANAGER/container-executor.cfg to /var/lib/yarn-ce/etc/hadoop/container-executor.cfg. [18/Jul/2020 17:05:05 +0000] 10596 MainThread __init__ INFO Copied special file /opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop-yarn/bin/container-executor to /var/lib/yarn-ce/bin/container-executor. [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO [136-yarn-NODEMANAGER] Evaluating resource: [{u'path': u'/data0/yarn/nm', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hadoop', u'user': u'yarn', u'mode': 493}, {u'path': u'/data0/yarn/container-logs', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hadoop', u'user': u'yarn', u'mode': 493}, {u'path': u'/var/lib/hadoop-yarn/yarn-nm-recovery', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hadoop', u'user': u'yarn', u'mode': 493}, {u'path': u'/var/log/hadoop-yarn', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hadoop', u'user': u'yarn', u'mode': 509}, {u'path': u'/var/log/hadoop-yarn/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'hadoop', u'user': u'yarn', u'mode': 493}] [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO reading limits: {u'limit_memlock': None, u'limit_fds': None} [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO [136-yarn-NODEMANAGER] Launching process. one-off False, command yarn/yarn.sh, args [u'nodemanager'] [18/Jul/2020 17:05:05 +0000] 10596 MainThread agent INFO Triggering supervisord update. [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO Begin audit plugin refresh [18/Jul/2020 17:05:05 +0000] 10596 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for yarn-NODEMANAGER with count 1 pipelines names ['']. [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO Begin metadata plugin refresh [18/Jul/2020 17:05:05 +0000] 10596 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for yarn-NODEMANAGER with count 1 pipelines names ['']. [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO Begin profile plugin refresh [18/Jul/2020 17:05:05 +0000] 10596 MainThread navigator_plugin INFO Scheduling a refresh for Profile Plugin for yarn-NODEMANAGER with count 1 pipelines names ['TP']. [18/Jul/2020 17:05:05 +0000] 10596 MainThread __init__ INFO Instantiating generic monitor for service YARN and role NODEMANAGER [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO Begin monitor refresh. [18/Jul/2020 17:05:05 +0000] 10596 MainThread abstract_monitor INFO Refreshing GenericMonitor YARN-NODEMANAGER for None [18/Jul/2020 17:05:05 +0000] 10596 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x7fa3452d1650>,) [18/Jul/2020 17:05:05 +0000] 10596 MainThread process INFO Daemon refresh complete for process 136-yarn-NODEMANAGER. [18/Jul/2020 17:05:07 +0000] 10596 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for yarn-NODEMANAGER with count 0 pipelines names []. [18/Jul/2020 17:05:07 +0000] 10596 Audit-Plugin navigator_plugin INFO Piplines updated for Audit Plugin: [] [18/Jul/2020 17:05:07 +0000] 10596 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for yarn-NODEMANAGER with count 0 pipelines names []. [18/Jul/2020 17:05:07 +0000] 10596 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for yarn-NODEMANAGER with count 0 pipelines names []. [18/Jul/2020 17:05:07 +0000] 10596 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [18/Jul/2020 17:05:07 +0000] 10596 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for yarn-NODEMANAGER with count 0 pipelines names []. [18/Jul/2020 17:05:07 +0000] 10596 Profile-Plugin navigator_plugin INFO stopping Profile Plugin for yarn-NODEMANAGER with count 0 pipelines names []. [18/Jul/2020 17:05:07 +0000] 10596 Profile-Plugin navigator_plugin INFO Piplines updated for Profile Plugin: [] [18/Jul/2020 17:05:07 +0000] 10596 Profile-Plugin navigator_plugin INFO Refreshing Profile Plugin for yarn-NODEMANAGER with count 0 pipelines names []. [18/Jul/2020 17:06:05 +0000] 10596 MonitorDaemon-Scheduler __init__ INFO Monitor expired: ('GenericMonitor YARN-NODEMANAGER for CD-YARN-PbpthEaX-NODEMANAGER-f2098a33ab399cc2ac267af283ff7430',)</LI-CODE>       Error that shows connection refused since NodeManager not running             <LI-CODE lang="markup">10596 Monitor-GenericMonitor throttling_logger ERROR (61 skipped) Error fetching metrics at 'http://us-east-1a-test-east-cdh-ta sknode4241.throtle-test.internal:8042/jmx' Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/generic/metric_collectors.py", line 200, in _collect_and_pa rse_and_return self._adapter.safety_valve)) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/url_util.py", line 223, in urlopen_with_retry_on_authentication_err ors return function() File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/generic/metric_collectors.py", line 217, in _open_url password=self._password_value) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/url_util.py", line 70, in urlopen_with_timeout return opener.open(url, data, timeout) File "/usr/lib64/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/usr/lib64/python2.7/urllib2.py", line 449, in _open '_open', req) File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open raise URLError(err) URLError: <urlopen error [Errno 111] Connection refused></LI-CODE>       Here is the log from supervisord.log file       <LI-CODE lang="markup">2020-07-18 17:05:05,228 INFO spawned: '136-yarn-NODEMANAGER' with pid 48155 2020-07-18 17:05:26,039 INFO success: 136-yarn-NODEMANAGER entered RUNNING state, process has stayed up for > than 20 seconds (startsecs)</LI-CODE>       Can any one help me in fix the issue?   Thanks & Regards,   Nagamalleswara. Fri, 16 Sep 2022 14:38:20 GMT Nagamalleswara 2022-09-16T14:38:20Z