Background: first time installing hadoop on a cluster single-handedly. AWS 3 instances, Ubuntu 16, 1 XL node 2 L nodes, all with 40 Gb storage each. Latest version of HDP 2.6.4. Installed successfully, started ambari successfully, logged into web admin portal ... Followed all the steps in the documentation.
Problem: most services failed to start ("heartbeat lost"), 39 alerts. Tried starting services manually via the web portal, tried restarting the ambari server, did not help. Below are some of the alerts:
HDFS NameNode Web UI:
Connection failed to http://ec2-18-217-xxxx.us-east-2.compute.amazonaws.com:50070 (<urlopen error [Errno 111] Connection refused>)
Yarn App Timeline Web UI:
Connection failed to http://ec2-18-218-xxxx.us-east-2.compute.amazonaws.com:8188/ws/v1/timeline (<urlopen error [Errno 111] Connection refused>) CRIT
MapReduce2 History Server Process
Connection failed: [Errno 111] Connection refused to ec2-18-218-xxxx.us-east-2.compute.amazonaws.com:19888
... and many more
@Geoffrey Shelton Okot Reviewed the link. Thank you for taking a look at this.
Best answer in the link just provides a link to general AWS documentation on Elastic IP and VPC DNS. I did not find reasons nor solutions for the issue I detailed here.
I've assigned elastic IPs to my cluster prior to the installation of the Ambari. Installation completed with no issues. Services are not starting due to failed connections, some of which are detailed in my original post. Same elastic IPs have been assigned to the cluster since its launch.
Is there a way to trouble-shoot these connections? Why the setup did not configure these connections automatically?
Is there a way/steps to verify that everything is properly configured on the cluster for the above connections to work?
@Geoffrey Shelton Okot thanks for responding and giving the pointers.
I did configure and test the passwordless connect between the hosts in the cluster. I can ssh to any host, from any host on the cluster without entering the password.
I did folllow the procedure from the installation guide for setting up /etc/hosts, but I will post the contents of /etc/hosts later today.
Just to eliminate again some doubts did you install the ambari-agents using ambari or manually if the later can you check that the ambari-agent.ini has the correct entry for the ambari-server (FQDN)?
In your /etc/hosts are you using the private of public IP's can the ip's/hostnames be resolved by DNS ?
I've installed everything (assuming agents as well) using ambari. I have not done any manual setup, to my knowledge. Below are the contents the /etc/hosts and /etc/ambari-agent/conf/ambari-agent.ini
Contents of /etc/hosts (I've masked last two numbers here with x for security):
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
Contents of /etc/ambari-agent/conf/ambari-agent.ini (host name below is set to the aws private IP that the 1st elastic IP in /etc/hosts above is pointing to, 18.217.xxx.xx, I have not set it anywhere myself, so this must have been determined by ambari setup process automatically. What do you recommend? Should I manually change it to the elastic IP from /etc/hosts, on each node? Any other files to check?):
; this option apply only for Agent communication
; 0 - unlimited
Changed ambari-agent.ini on all the nodes, replaced the private IP of the master node with elastic IP of the master node. Stopped and started the ambari server. This did not help, same issue continues. Issue is not resolved.
Any other checks/trouble-shooting options?