question Re: Ambari Agent failed to start in manual installation in Archives of Support Questions (Read Only)

Ambari Agent failed to start in manual installation

munnyrahul — Wed, 15 Feb 2017 16:59:56 GMT

I am installing 4 node cluster using ambari. I installed ambari server and ambari agent on master node. In addition to that i have installed ambari agents on 2 slave nodes. Ambari server and ambari agents on master node is running.But i am unable to start ambari-agent on my slave nodes due to the error "Ambari Agent start fails with local variable 'server_url' referenced before assignment". This seems like jira bug but it has already been fixed.

https://issues.apache.org/jira/browse/AMBARI-19246

I am using ambari version 2.4.2

Could anyone please help me in starting agents on slave nodes?

Re: Ambari Agent failed to start in manual installation

sshimpi — Wed, 15 Feb 2017 17:14:34 GMT

@rahul gulati

Yes, its a bug also reported in HDP JIRA - https://hortonworks.jira.com/browse/BUG-70990 [Internal link for reference]

This will be fixed in up coming Ambari Release version 2.5.0.

If you are looking to get a patch to Fix this issue, please reach out to Hortonworks Support team to get Subscription.

Re: Ambari Agent failed to start in manual installation

sshimpi — Wed, 15 Feb 2017 17:16:59 GMT

@rahul gulati

Check if the below think works and resolve the issue -

Suspected the issue is with snmpd service not installed or installed and not running. Execute: 1) service snmpd start If this does not recognize the service then snmpd needs to be installed and proceed with remaining steps. 2) yum -y install net-snmp net-snmp-utils 3) service snmpd start

Re: Ambari Agent failed to start in manual installation

munnyrahul — Wed, 15 Feb 2017 17:17:16 GMT

@Sagar Shimpi

Thanks for replying. Then i want to check how to register host in case of ambari installation if manual installation is not working? I am stuck at registering host since i cannot install and start ambari agents on slave nodes.

Thanks

Re: Ambari Agent failed to start in manual installation

jsensharma — Wed, 15 Feb 2017 17:19:13 GMT

@rahul gulati

Although the workaround is quite simple. Just edit the file "/usr/lib/python2.6/site-packages/ambari_agent/main.py"

Just change the line "server_ip = socket.gethostbyname(server_hostname)" as mentioned in:

https://git1-us-west.apache.org/repos/asf?p=ambari.git;a=commitdiff;h=a43837a35536096224a77356a4049ea5b01a8bdc;hp=8b3ec80cef1ac453b53b24cd0e0efa944cb45c89

This need to be done on all the ambari-agent hosts and then followed by "ambari-agent restart"

NOTE: It should help only if your issue is exactly same as the one mentioned in :

https://issues.apache.org/jira/browse/AMBARI-19246

Re: Ambari Agent failed to start in manual installation

munnyrahul — Wed, 15 Feb 2017 17:36:39 GMT

@Jay SenSharma

Thanks for the answer. I am able to start the ambari agent successfully on all the 3 nodes. Thanks to you. But while registering the agents from ambari UI i am getting Registration failed for slave nodes although master is able to register successfully. Any idea why? I am not able to see any other logs.

Thanks

Re: Ambari Agent failed to start in manual installation

jsensharma — Wed, 15 Feb 2017 17:41:44 GMT

@rahul gulati

Good to know that your original issue is resolved.

Can you please share more details about your latest error?

1. ambari-server.log snippet

2. Screenshot of the error.

Few things you might want to check is

1. If you are able to do password less ssh to those hosts. That might not be needed though.

2. If ambari server is able to recognize all the agents hosts using their "hostname -f" (FQDN output)

Re: Ambari Agent failed to start in manual installation

munnyrahul — Wed, 15 Feb 2017 18:02:45 GMT

@Jay SenSharma

actually it was not able to establish connection to ambari server hostname. So i changed the value from hostname to IP address in ambari.ini file. And it worked. But i wanted to ask one thing that i kept host name in master node ambari.ini file where ambari server is installed and it worked in first go but in slave nodes ambari.ini file it worked wit IP. Any specific reasons for that?

And once again that for the above answer. It worked like charm. Could not be possible to wait till 2.5 release. 🙂

Thanks

Re: Ambari Agent failed to start in manual installation

jsensharma — Wed, 15 Feb 2017 18:58:29 GMT

@rahul gulati

Good to know that it worked well after fixing the hostname.

Ambari (and also HDP) is very much dependent on the FQDN (hostname -f) so every host must be able to recognize then using the FQDN (instead of IP). Even ambari agent script uses the same approach to get the hostname of it's own.

host_name = socket.gethostname().lower()
host_ip = socket.gethostbyname(host_name)
host_fqdn = socket.getfqdn().lower()
fqdn_ip = socket.gethostbyname(host_fqdn)
return host_ip == fqdn_ip

See:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/_edit_the_host_file.html

Re: Ambari Agent failed to start in manual installation

munnyrahul — Wed, 15 Feb 2017 21:44:54 GMT

@Jay SenSharma

My ambari-agents are successfully registered with ambari-server but i am seeing a warning which says that hostname resolution issue in which it says not all the hosts could resolve hostname of other hosts.

Although i have added the IP addresses and hostnames of all the hosts in /etc/hosts files of all the 4 nodes. But still it is not able to resolve. Anything else to do?

Below is my example /etc/hosts file on edge node(where abcd, etf, sco, jfjf are hostnames of edge, master, slave1 and slave 2 nodes resp). Same goes with all other nodes as well

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

1.2.3.4 abcd

3.4.5.6 etf

7.8.9.1 sco

10.11.12.13 jfjf

Please note that the ambari.ini file on slave1, slave2 and edge node has IP address as hostname while master node file has hostname of masternode. This is how it is setup and is working. Because i tried to register ambari agents by giving hostname in ambari.ini files of slaves and edge node but i was getting failure but master ambari agent was able to register with ambari server with fqdn.

Re: Ambari Agent failed to start in manual installation

jsensharma — Wed, 15 Feb 2017 23:03:07 GMT

@rahul gulati

Are you sure that when you are running the following command on those hosts individually then you are getting the correct FQDN?

hostname -f

- Ambari agent sends registration request to ambari-server every time when agent is restarted. In the registration request it sends the FQDN information of it's own as well. Agent uses the following approach to findout the FQDN using python. So please check if all your hosts are returning correct hostname

python -c "import socket; print socket.getfqdn()"

Also in local system it might get using which is not actually FQDN as above. So local agent might behave bit differently then remote.

python -c "import socket; print socket.gethostname()"