Ambari agent registration of HDF cluster fails inspite of exitcode 0. Setup of RHEL on MS Azure.

[#1]

==========================

Creating target directory...
==========================

Command start time 2018-05-16 06:08:52
chmod: cannot access ‘/var/lib/ambari-agent/data’: No such file or directory

Warning: Permanently added 'mtvm6.eastus.cloudapp.azure.com,40.117.251.23' (ECDSA) to the list of known hosts.
Connection to mtvm6.eastus.cloudapp.azure.com closed.
SSH command execution finished
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:52

==========================
Copying ambari sudo script...
==========================

Command start time 2018-05-16 06:08:52

scp /var/lib/ambari-server/ambari-sudo.sh
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:53

==========================
Copying common functions script...
==========================

Command start time 2018-05-16 06:08:53

scp /usr/lib/python2.6/site-packages/ambari_commons
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:53

==========================
Copying create-python-wrap script...
==========================

Command start time 2018-05-16 06:08:53

scp /var/lib/ambari-server/create-python-wrap.sh
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:54

==========================
Copying OS type check script...
==========================

Command start time 2018-05-16 06:08:54

scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:54

==========================
Running create-python-wrap script...
==========================

Command start time 2018-05-16 06:08:54

Connection to mtvm6.eastus.cloudapp.azure.com closed.
SSH command execution finished
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:55

==========================
Running OS type check...
==========================

Command start time 2018-05-16 06:08:55
Cluster primary/cluster OS family is redhat7 and local/current OS family is redhat7

Connection to mtvm6.eastus.cloudapp.azure.com closed.
SSH command execution finished
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:55

==========================
Checking 'sudo' package on remote host...
==========================

Command start time 2018-05-16 06:08:55
sudo-1.8.19p2-11.el7_4.x86_64

Connection to mtvm6.eastus.cloudapp.azure.com closed.
SSH command execution finished
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:56

==========================
Copying repo file to 'tmp' folder...
==========================

Command start time 2018-05-16 06:08:56

scp /etc/yum.repos.d/ambari.repo
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:57

==========================
Moving file to repo dir...
==========================

Command start time 2018-05-16 06:08:57

Connection to mtvm6.eastus.cloudapp.azure.com closed.
SSH command execution finished
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:57

==========================
Changing permissions for ambari.repo...
==========================

Command start time 2018-05-16 06:08:57

Connection to mtvm6.eastus.cloudapp.azure.com closed.
SSH command execution finished
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:57

==========================
Copying setup script file...
==========================

Command start time 2018-05-16 06:08:57

scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:08:58

==========================
Running setup agent script...
==========================

Command start time 2018-05-16 06:08:58
("INFO 2018-05-16 06:09:18,024 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 06:09:18,024 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 06:09:18,024 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 06:09:18,025 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-05-16 06:09:18,027 DataCleaner.py:120 - Data cleanup started
INFO 2018-05-16 06:09:18,027 DataCleaner.py:122 - Data cleanup finished
INFO 2018-05-16 06:09:18,028 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'mtvm6.eastus.cloudapp.azure.com' using socket.getfqdn().
INFO 2018-05-16 06:09:18,035 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-05-16 06:09:18,038 main.py:437 - Connecting to Ambari server at https://myhdf.eastus.cloudapp.azure.com:8440 (104.211.60.99)
INFO 2018-05-16 06:09:18,038 NetUtil.py:70 - Connecting to https://myhdf.eastus.cloudapp.azure.com:8440/ca
", None)
("INFO 2018-05-16 06:09:18,024 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 06:09:18,024 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 06:09:18,024 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 06:09:18,025 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-05-16 06:09:18,027 DataCleaner.py:120 - Data cleanup started
INFO 2018-05-16 06:09:18,027 DataCleaner.py:122 - Data cleanup finished
INFO 2018-05-16 06:09:18,028 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'mtvm6.eastus.cloudapp.azure.com' using socket.getfqdn().
INFO 2018-05-16 06:09:18,035 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-05-16 06:09:18,038 main.py:437 - Connecting to Ambari server at https://myhdf.eastus.cloudapp.azure.com:8440 (104.211.60.99)
INFO 2018-05-16 06:09:18,038 NetUtil.py:70 - Connecting to https://myhdf.eastus.cloudapp.azure.com:8440/ca
", None)

Connection to mtvm6.eastus.cloudapp.azure.com closed.
SSH command execution finished
host=mtvm6.eastus.cloudapp.azure.com, exitcode=0
Command end time 2018-05-16 06:09:20

Registering with the server...
Registration with the server failed.

[#2]

@Matthias Tewordt

Can you create this directory and ensure the permissions are correct

mkdir -p /var/lib/ambari-agent/data

Then re-run the cluster setup

[#3]

a new try provided these logs:

- what does "Ambani-Agent received 15 signal, stopping ..." mean ?

- how can I best troubleshoot "NetUtil.py:101 - Failed to connect to https://myhdf.eastus.cloudapp.azure.com:8440/ca due to [Errno 4] Interrupted system call " ?

==========================
Running setup agent script...
==========================

Command start time 2018-05-16 14:32:42
("INFO 2018-05-16 14:32:44,985 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 14:32:44,993 HeartbeatHandlers.py:84 - Ambari-agent received 15 signal, stopping...
WARNING 2018-05-16 14:32:44,993 NetUtil.py:101 - Failed to connect to https://myhdf.eastus.cloudapp.azure.com:8440/ca due to [Errno 4] Interrupted system call  
WARNING 2018-05-16 14:32:44,993 NetUtil.py:124 - Server at https://myhdf.eastus.cloudapp.azure.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-16 14:32:44,994 HeartbeatHandlers.py:116 - Stop event received
INFO 2018-05-16 14:32:44,994 NetUtil.py:130 - Stop event received
INFO 2018-05-16 14:32:44,994 ExitHelper.py:56 - Performing cleanup before exiting...
INFO 2018-05-16 14:32:44,994 ExitHelper.py:70 - Cleanup finished, exiting with code:0 

INFO 2018-05-16 14:32:45,016 main.py:283 - Agent died gracefully, exiting.

[#4]

thanks Geoffrey for your suggestion. I introduced mkdir -p /var/lib/ambari-agent/data with chmod 770 for all 3 nodes. It didn't really help. This is is the head of the "Running Set up script" log of mtvm6:

again a mtvm6 failed to connect to myhdf

==========================
Running setup agent script...
==========================

Command start time 2018-05-16 18:18:03
("INFO 2018-05-16 18:18:05,472 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 18:18:05,478 HeartbeatHandlers.py:84 - Ambari-agent received 15 signal, stopping...
WARNING 2018-05-16 18:18:05,478 NetUtil.py:101 - Failed to connect to https://myhdf.eastus.cloudapp.azure.com:8440/ca due to [Errno 4] Interrupted system call  
WARNING 2018-05-16 18:18:05,478 NetUtil.py:124 - Server at https://myhdf.eastus.cloudapp.azure.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-16 18:18:05,478 HeartbeatHandlers.py:116 - Stop event received
INFO 2018-05-16 18:18:05,478 NetUtil.py:130 - Stop event received
INFO 2018-05-16 18:18:05,478 ExitHelper.py:56 - Performing cleanup before exiting...

WARNING 2018-05-16 18:18:06,261 NetUtil.py:124 - Server at https://myhdf.eastus.cloudapp.azure.com:8440 is not reachable, sleeping for 10 seconds...

[#5]

@Matthias Tewordt

Could you take a backup of cert-verification.cfg

cp  /etc/python/cert-verification.cfg /etc/python/cert-verification.cfg.bak 

Then update /etc/python/cert-verification.cfg to have verify=disable or please create this file

sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg 

Retry and let me know

[#6]

Geoffrey, I have updated the cert-verification.cfg as proposed. Still fails with same error messages:

Command start time 2018-05-16 20:33:33
("INFO 2018-05-16 20:33:35,736 main.py:145 - loglevel=logging.INFO
INFO 2018-05-16 20:33:35,742 HeartbeatHandlers.py:84 - Ambari-agent received 15 signal, stopping...
WARNING 2018-05-16 20:33:35,742 NetUtil.py:101 - Failed to connect to https://myhdf.eastus.cloudapp.azure.com:8440/ca due to [Errno 4] Interrupted system call  
WARNING 2018-05-16 20:33:35,743 NetUtil.py:124 - Server at https://myhdf.eastus.cloudapp.azure.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-16 20:33:35,743 HeartbeatHandlers.py:116 - Stop event received
INFO 2018-05-16 20:33:35,743 NetUtil.py:130 - Stop event received 

INFO 2018-05-16 20:33:35,743 ExitHelper.py:56 - Performing cleanup before exiting...

[#7]

@Matthias Tewordt

If you have only 3 nodes in your cluster then we could try the manual registration. But before we go that direction did you follow the documented step of preparing the environment

• Set Up Password-less SSH
• Enable NTP on the Cluster and on the Browser Host
• Check DNS and NSCD
• Configuring iptables
• Disable SELinux and PackageKit and check the umask Value

• Enable NTP on Your Cluster
• Disable IPTables
• THP

Just to be sure..... If you accomplished the above then, adapt the below for your OS specific. I assume your repos were correctly set and are accessible, you can validate with

# yum repolist

You should be able to see HDP,HDP-UTILS and Ambari repos

Install ambari-agent on all nodes include the Ambari node

# yum install -y ambari-agent

Edit the ambari-agent.ini see example located at /etc/ambari-agent/conf on all the hosts and

[server] 
hostname={Ambari_FQDN} 
url_port=8440 
secured_url_port=8441 
connect_retry_delay=10 
max_reconnect_retry_delay=30

This should hostname should be the Ambari server FQDN

Start the agent

# ambari-agent start

The above should be done on all the 3 nodes and ensure they started successfully

Log on the Ambari UI and follow the previous steps, now in the host registration process chose manual after, you won't need the ssh generated key

The process should complete with GREEN and there you can proceed with your deployment

Please revert

[#8]

Geoffrey, thanks a lot for your help. I have followed every step in your list and then chose manual registration on the amber ui. It failed again with the following note:

Registering with the server... Registration with the server failed.

[#9]

Geoffrey, I noted that a manual install on the master actually was successful. Can we conclude that there are connectivity problems ? Ping -c3 10.0.0.5 etc works but I could not successfully telnet from a node to the master. What kind of checks does it need from your perpective beyond successful ssh root@node ?

Thanks again, Matthias

[#10]

Definitely a conne tivity problem.

Check your /etc/host entry and the DNS resolution