unable to add the datanode to the cluster

[#1]

[10/Nov/2023 13:52:40 +0000] 8759 MainThread tmpfs INFO Reusing mounted tmpfs at /run/cloudera-scm-agent/process
[10/Nov/2023 13:52:41 +0000] 8759 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 1)
[10/Nov/2023 13:52:41 +0000] 8759 MainThread agent INFO Supervisor version: 3.0, pid: 8778
[10/Nov/2023 13:52:41 +0000] 8759 MainThread agent INFO Successfully connected to supervisor
[10/Nov/2023 13:52:41 +0000] 8759 MainThread status_server INFO Using maximum impala profile bundle size of 1073741824 bytes.
[10/Nov/2023 13:52:41 +0000] 8759 MainThread status_server INFO Using maximum stacks log bundle size of 1073741824 bytes.
[10/Nov/2023 13:52:41 +0000] 8759 MainThread _cplogging INFO [10/Nov/2023:13:52:41] ENGINE Bus STARTING
[10/Nov/2023 13:52:41 +0000] 8759 MainThread _cplogging INFO [10/Nov/2023:13:52:41] ENGINE Started monitor thread '_TimeoutMonitor'.
[10/Nov/2023 13:52:42 +0000] 8759 MainThread _cplogging INFO [10/Nov/2023:13:52:42] ENGINE Serving on eyrkonaepradn01:9000
[10/Nov/2023 13:52:42 +0000] 8759 MainThread _cplogging INFO [10/Nov/2023:13:52:42] ENGINE Bus STARTED
[10/Nov/2023 13:52:42 +0000] 8759 MainThread __init__ INFO New monitor: (<cmf.monitor.host.HostMonitor object at 0x7ff7eb02d890>,)
[10/Nov/2023 13:52:42 +0000] 8759 MonitorDaemon-Scheduler __init__ INFO Monitor ready to report: ('HostMonitor',)
[10/Nov/2023 13:52:42 +0000] 8759 MainThread agent INFO Setting default socket timeout to 45
[10/Nov/2023 13:52:42 +0000] 8759 Monitor-HostMonitor network_interfaces INFO NIC iface eth0 doesn't support ETHTOOL (95)
[10/Nov/2023 13:52:42 +0000] 8759 Monitor-HostMonitor network_interfaces INFO NIC iface eth1 doesn't support ETHTOOL (95)
[10/Nov/2023 13:52:42 +0000] 8759 Monitor-HostMonitor network_interfaces INFO NIC iface eth2 doesn't support ETHTOOL (95)
[10/Nov/2023 13:52:42 +0000] 8759 MainThread heartbeat_tracker INFO HB stats (seconds): num:1 LIFE_MIN:0.02 min:0.02 mean:0.02 max:0.02 LIFE_MAX:0.02

Getting above error when trying to add a data node to the cluster, 

Anyone can help?

[#2]

@soychago  i have pasted the log above

[#3]

@sharonsony thanks for the copy/pasting the log excerpt.

From the above output, we cannot spot any datanode addition activity.

We do see though, that the agent fails to communicate to the supervisord getting connection refused:

[10/Nov/2023 13:52:40 +0000] 8759 MainThread agent ERROR Failed to connect to previous supervisor.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/agent.py", line 2146, in find_or_start_supervisor
self.get_supervisor_process_info()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.1-py2.7.egg/cmf/agent.py", line 2290, in get_supervisor_process_info
self.identifier = self.supervisor_client.supervisor.getIdentification()
File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request
verbose=self.__verbose
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request
self.connection.request('POST', handler, request_body, self.headers)
File "/usr/lib64/python2.7/httplib.py", line 1041, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 843, in send
self.connect()
File "/usr/lib64/python2.7/httplib.py", line 824, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
raise err
error: [Errno 111] Connection refused

 I think you can try to hard-stop and restart the agents (since you're in CDH 5.16.x):

Starting, Stopping, and Restarting Cloudera Manager Agents | 5.16.x | Cloudera Documentation

Warning: The hard_stop and hard_restart commands kill all running managed service processes on the host(s) where the command is run.

[#4]

@soychago hard-stop and restarting the agents are done already ...But still not abl to add the data node to the cluster.... i have added other 4 nodes , but this node i am talking about cannot be added to the cluster.