Ambari Agent failed to start in manual installation

[#1]

I am installing 4 node cluster using ambari. I installed ambari server and ambari agent on master node. In addition to that i have installed ambari agents on 2 slave nodes. Ambari server and ambari agents on master node is running.But i am unable to start ambari-agent on my slave nodes due to the error "Ambari Agent start fails with local variable 'server_url' referenced before assignment". This seems like jira bug but it has already been fixed.

https://issues.apache.org/jira/browse/AMBARI-19246

I am using ambari version 2.4.2

Could anyone please help me in starting agents on slave nodes?

[#2]

@rahul gulati

Yes, its a bug also reported in HDP JIRA - https://hortonworks.jira.com/browse/BUG-70990 [Internal link for reference]

This will be fixed in up coming Ambari Release version 2.5.0.

If you are looking to get a patch to Fix this issue, please reach out to Hortonworks Support team to get Subscription.

[#3]

@rahul gulati

Check if the below think works and resolve the issue -

Suspected the issue is with snmpd service not installed or installed and not running. Execute: 1) service snmpd start If this does not recognize the service then snmpd needs to be installed and proceed with remaining steps. 2) yum -y install net-snmp net-snmp-utils 3) service snmpd start

[#4]

@Sagar Shimpi

Thanks for replying. Then i want to check how to register host in case of ambari installation if manual installation is not working? I am stuck at registering host since i cannot install and start ambari agents on slave nodes.

Thanks

[#5 · Accepted solution]

@rahul gulati

Although the workaround is quite simple. Just edit the file "/usr/lib/python2.6/site-packages/ambari_agent/main.py"

Just change the line "server_ip = socket.gethostbyname(server_hostname)" as mentioned in:

https://git1-us-west.apache.org/repos/asf?p=ambari.git;a=commitdiff;h=a43837a35536096224a77356a4049e...

This need to be done on all the ambari-agent hosts and then followed by "ambari-agent restart"

.

NOTE: It should help only if your issue is exactly same as the one mentioned in :

https://issues.apache.org/jira/browse/AMBARI-19246

[#6]

@Jay SenSharma

Thanks for the answer. I am able to start the ambari agent successfully on all the 3 nodes. Thanks to you. But while registering the agents from ambari UI i am getting Registration failed for slave nodes although master is able to register successfully. Any idea why? I am not able to see any other logs.

Thanks

[#7]

@rahul gulati

Good to know that your original issue is resolved.

Can you please share more details about your latest error?

1. ambari-server.log snippet

2. Screenshot of the error.

Few things you might want to check is

1. If you are able to do password less ssh to those hosts. That might not be needed though.

2. If ambari server is able to recognize all the agents hosts using their "hostname -f" (FQDN output)

[#8]

@Jay SenSharma

actually it was not able to establish connection to ambari server hostname. So i changed the value from hostname to IP address in ambari.ini file. And it worked. But i wanted to ask one thing that i kept host name in master node ambari.ini file where ambari server is installed and it worked in first go but in slave nodes ambari.ini file it worked wit IP. Any specific reasons for that?

And once again that for the above answer. It worked like charm. Could not be possible to wait till 2.5 release. 🙂

Thanks

[#9]

@rahul gulati

Good to know that it worked well after fixing the hostname.

Ambari (and also HDP) is very much dependent on the FQDN (hostname -f) so every host must be able to recognize then using the FQDN (instead of IP). Even ambari agent script uses the same approach to get the hostname of it's own.

host_name = socket.gethostname().lower()
host_ip = socket.gethostbyname(host_name)
host_fqdn = socket.getfqdn().lower()
fqdn_ip = socket.gethostbyname(host_fqdn)
return host_ip == fqdn_ip

.

See:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/_edit_the_host...

[#10]

@Jay SenSharma

My ambari-agents are successfully registered with ambari-server but i am seeing a warning which says that hostname resolution issue in which it says not all the hosts could resolve hostname of other hosts.

Although i have added the IP addresses and hostnames of all the hosts in /etc/hosts files of all the 4 nodes. But still it is not able to resolve. Anything else to do?

Below is my example /etc/hosts file on edge node(where abcd, etf, sco, jfjf are hostnames of edge, master, slave1 and slave 2 nodes resp). Same goes with all other nodes as well

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

1.2.3.4 abcd

3.4.5.6 etf

7.8.9.1 sco

10.11.12.13 jfjf

Please note that the ambari.ini file on slave1, slave2 and edge node has IP address as hostname while master node file has hostname of masternode. This is how it is setup and is working. Because i tried to register ambari agents by giving hostname in ambari.ini files of slaves and edge node but i was getting failure but master ambari agent was able to register with ambari server with fqdn.