Ambari agent not able to register with the server?

[#1]

INFO 2018-08-18 12:38:30,061 DataCleaner.py:120 - Data cleanup started INFO 2018-08-18 12:38:30,061 DataCleaner.py:122 - Data cleanup finished INFO 2018-08-18 12:38:30,073 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2018-08-18 12:38:30,076 main.py:437 - Connecting to Ambari server at https://localhost:8440 (127.0.0.1) INFO 2018-08-18 12:38:30,076 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:38:30,077 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:38:30,077 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:38:40,077 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:38:40,078 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:38:40,078 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:38:50,078 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:38:50,079 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:38:50,079 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:39:00,080 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:39:00,080 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:39:00,081 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:39:10,081 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:39:10,082 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:39:10,082 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:39:20,082 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:39:20,083 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:39:20,083 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:39:30,083 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:39:30,084 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:39:30,084 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:39:40,089 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:39:40,090 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:39:40,090 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:39:50,091 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:39:50,091 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:39:50,092 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:40:00,092 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:40:00,093 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:40:00,093 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:40:10,093 main.py:437 - Connecting to Ambari server at https://localhost:8440 (127.0.0.1) INFO 2018-08-18 12:40:10,093 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:40:10,094 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:40:10,094 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:40:20,094 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:40:20,095 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:40:20,095 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:40:30,096 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:40:30,097 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:40:30,097 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds... INFO 2018-08-18 12:40:40,097 NetUtil.py:70 - Connecting to https://localhost:8440/ca WARNING 2018-08-18 12:40:40,098 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused WARNING 2018-08-18 12:40:40,098 NetUtil.py:124 - Server at https://localhost:8440 is not reachable, sleeping for 10 seconds...

[#2]

@venu gopal

As we see the error as following:

WARNING 2018-08-18 12:39:00,080 NetUtil.py:101 - Failed to connect to https://localhost:8440/ca due to [Errno 111] Connection refused 

Which indicates that you might not have configured the ambari server host name correctly inside the "/etc/ambari-agent/conf/ambari-agent.ini" file (that why it is trying to connect to a localhost) to point to your Ambari Server Hostname.

# grep 'hostname' /etc/ambari-agent/conf/ambari-agent.ini
hostname=ambariserver.example.com

Please make sure that the ambari agent.ini file is using correct ambari server hostname.

Also from ambari agent host you should be able to access the ambari server host & port using the following command to verify if there is any Port access / firewall issue. So from amabri Agent host please verify if it can connect to ambari server?

# nc -v $AMBARI_HOST 8440

Also on the ambari server host please verify it's correct hostname and also if it has opened port 8440 or not? And also if the IPtables/Firewall is disabled or not?

# hostname -f
# service iptables stop

.

[#3]

@Jay Kumar SenSharma

I have a similar issue and I am getting NetUtil.py:92 - Failed to connect to https://master1.hadoop.com:8440/ca due to [Errno 111] Connection refused

I have gone through your suggestions and verified my settings. I have all the right settings in place like firewalls disabled, ambari-agent.ini file has right server name etc. I can also ping master node ip address from worker node, and I can see the responses from master. It means that both the nodes are connected.

I have also added the following to ambari-agen.ini file

force_https_protocol=PROTOCOL_TLSv1_2

I have also made the changes in /etc/python/cert-verification.cfg

verify=disable

But when I tried the command "nc -v $AMBARI_HOST 8440 " as mentioned by you, then I found that the command works in master node but not in worker node. It gives me the error message as "Connection refused". So it means that somewhere my set up is missing something.

Can you suggest a solution for this problem. Please let me know if you need more details about it?