Install HDP 2.5 using Ambari 2.4 on AWS Fails

[#1]

Hi,

Except for the instance that has Ambari Server and Agent running I can't install HDP 2.5 using Ambari 2.5. I've installed the agents manually successfully. I'm not sure I ave the /etc/hosts file right. I'm using <private IP> <pribate DNS> and the <private DNS> in the Ambari GUI. But they fail to install all except the instance that has Ambari Server runnning.

I'm using Centos 7. Any ideas?

[#2]

This is the error - Registering with the server...

Registration with the server failed.

[#3]

@David Sheard Can you ping other nodes from Ambari nodes ?

[#4]

yes I can even ssh key-less

[#5]

@David Sheard

Did you tried using public FQDN in hosts tab while registering host in Ambari UI.. Let me know if that works ?

[#6]

Tried both public and private in GUI also in hosts file with no luck

[#7]

Can you pass the full log for the host registration failed ?

[#8]

@David Sheard

I think you need to understand the AWS networking.

Please check out the Amazon EC2 Instance IP Addressing

[#9]

Still having issue

==========================
Creating target directory...
==========================

Command start time 2016-12-28 04:19:41

Connection to ec2-54-206-122-4.ap-southeast-2.compute.amazonaws.com closed.
SSH command execution finished
host=ec2-xxxxxxx.ap-southeast-2.compute.amazonaws.com, exitcode=0
Command end time 2016-12-28 04:19:41

==========================
Copying ambari sudo script...
==========================

Command start time 2016-12-28 04:19:41

scp /var/lib/ambari-server/ambari-sudo.sh
host=ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com, exitcode=0
Command end time 2016-12-28 04:19:41

==========================
Copying common functions script...
==========================

Command start time 2016-12-28 04:19:41

scp /usr/lib/python2.6/site-packages/ambari_commons
host=ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com, exitcode=0
Command end time 2016-12-28 04:19:42

==========================
Copying OS type check script...
==========================

Command start time 2016-12-28 04:19:42

scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
host=ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com, exitcode=0
Command end time 2016-12-28 04:19:42

==========================
Running OS type check...
==========================

Command start time 2016-12-28 04:19:42
Cluster primary/cluster OS family is redhat7 and local/current OS family is redhat7

Connection to ec2-xxxxx.ap-southeast-2.compute.amazonaws.com closed.
SSH command execution finished
host=ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com, exitcode=0
Command end time 2016-12-28 04:19:42

==========================
Checking 'sudo' package on remote host...
==========================

Command start time 2016-12-28 04:19:42

Connection to ecxxxxxx.ap-southeast-2.compute.amazonaws.com closed.
SSH command execution finished
host=ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com, exitcode=0
Command end time 2016-12-28 04:19:43

==========================
Copying required files...
Ambari repo file not found: /etc/yum.repos.d/ambari.repo
==========================
Copying setup script file...
==========================

Command start time 2016-12-28 04:19:43

scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
host=ec2-xxxxxxx.ap-southeast-2.compute.amazonaws.com, exitcode=0
Command end time 2016-12-28 04:19:43

==========================
Running setup agent script...
==========================

Command start time 2016-12-28 04:19:43
('INFO 2016-12-28 04:14:20,333 NetUtil.py:62 - Connecting to https://ec2-xxxxxxxxx.ap-southeast-2.compute.amazonaws.com:8440/ca
WARNING 2016-12-28 04:14:20,334 NetUtil.py:93 - Failed to connect to https://ec2-xxxxxxx.ap-southeast-2.compute.amazonaws.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-12-28 04:14:20,334 NetUtil.py:116 - Server at https://ec2-xxxxxxxx.ap-southeast-2.compute.amazonaws.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-12-28 04:14:20,334 HeartbeatHandlers.py:115 - Stop event received
INFO 2016-12-28 04:14:20,334 NetUtil.py:122 - Stop event received
INFO 2016-12-28 04:14:20,334 ExitHelper.py:53 - Performing cleanup before exiting...
INFO 2016-12-28 04:14:20,334 ExitHelper.py:67 - Cleanup finished, exiting with code:0
INFO 2016-12-28 04:14:22,156 main.py:223 - Agent died gracefully, exiting.
INFO 2016-12-28 04:14:22,157 ExitHelper.py:53 - Performing cleanup before exiting...
INFO 2016-12-28 04:19:44,977 main.py:90 - loglevel=logging.INFO
INFO 2016-12-28 04:19:44,977 main.py:90 - loglevel=logging.INFO
INFO 2016-12-28 04:19:44,977 main.py:90 - loglevel=logging.INFO
INFO 2016-12-28 04:19:44,978 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-12-28 04:19:44,979 DataCleaner.py:120 - Data cleanup started
INFO 2016-12-28 04:19:44,980 DataCleaner.py:122 - Data cleanup finished
INFO 2016-12-28 04:19:45,022 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2016-12-28 04:19:45,023 main.py:349 - Connecting to Ambari server at https://ec2-xxxxxxxx.ap-southeast-2.compute.amazonaws.com:8440 (xxxxxxx)
INFO 2016-12-28 04:19:45,023 NetUtil.py:62 - Connecting to https://ec2-xxxxxxx.ap-southeast-2.compute.amazonaws.com:8440/ca
WARNING 2016-12-28 04:19:45,024 NetUtil.py:93 - Failed to connect to https://ec2-xxxxxxx.ap-southeast-2.compute.amazonaws.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-12-28 04:19:45,025 NetUtil.py:116 - Server at https://ec2-xxxxxxx.ap-southeast-2.compute.amazonaws.com:8440 is not reachable, sleeping for 10 seconds...
', None)
('INFO 2016-12-28 04:14:20,333 NetUtil.py:62 - Connecting to https://ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com:8440/ca
WARNING 2016-12-28 04:14:20,334 NetUtil.py:93 - Failed to connect to https://ec2-xxxxx.ap-southeast-2.compute.amazonaws.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-12-28 04:14:20,334 NetUtil.py:116 - Server at https://ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2016-12-28 04:14:20,334 HeartbeatHandlers.py:115 - Stop event received
INFO 2016-12-28 04:14:20,334 NetUtil.py:122 - Stop event received
INFO 2016-12-28 04:14:20,334 ExitHelper.py:53 - Performing cleanup before exiting...
INFO 2016-12-28 04:14:20,334 ExitHelper.py:67 - Cleanup finished, exiting with code:0
INFO 2016-12-28 04:14:22,156 main.py:223 - Agent died gracefully, exiting.
INFO 2016-12-28 04:14:22,157 ExitHelper.py:53 - Performing cleanup before exiting...
INFO 2016-12-28 04:19:44,977 main.py:90 - loglevel=logging.INFO
INFO 2016-12-28 04:19:44,977 main.py:90 - loglevel=logging.INFO
INFO 2016-12-28 04:19:44,977 main.py:90 - loglevel=logging.INFO
INFO 2016-12-28 04:19:44,978 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-12-28 04:19:44,979 DataCleaner.py:120 - Data cleanup started
INFO 2016-12-28 04:19:44,980 DataCleaner.py:122 - Data cleanup finished
INFO 2016-12-28 04:19:45,022 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2016-12-28 04:19:45,023 main.py:349 - Connecting to Ambari server at https://ec2-xxxxx.ap-southeast-2.compute.amazonaws.com:8440 (54.206.122.4)
INFO 2016-12-28 04:19:45,023 NetUtil.py:62 - Connecting to https://ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com:8440/ca
WARNING 2016-12-28 04:19:45,024 NetUtil.py:93 - Failed to connect to https://ec2-xxxxxx.ap-southeast-2.compute.amazonaws.com:8440/ca due to [Errno 111] Connection refused  
WARNING 2016-12-28 04:19:45,025 NetUtil.py:116 - Server at https://ec2-xxxxx.ap-southeast-2.compute.amazonaws.com:8440 is not reachable, sleeping for 10 seconds...
', None)

Connection to ec2-xxxxx.ap-southeast-2.compute.amazonaws.com closed.
SSH command execution finished
host=ec2-xxxxx.ap-southeast-2.compute.amazonaws.com, exitcode=0
Command end time 2016-12-28 04:19:47

Registering with the server...
Registration with the server failed.

[#10]

Checked https://community.hortonworks.com/questions/145/openssl-error-upon-host-registration.html

And I'm using java 8 on both.

Tried using ssh keys and also manual install of ambari-agent but no luck. I can get the server running and the agents on both servers but no luck via ambari installing HDP. I can see the agent startup when lauching from Ambari but then it can't connect to the server

[#11]

I notice it launches 2 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py scripts ??

[root@ec2-xxxxx ec2-user]# ps aux | grep ambari

root 16253 0.0 0.1 385276 17736 ? Sl 04:19 0:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start --expected-hostname=ec2-xxxxx.ap-southeast-2.compute.amazonaws.com

root 16344 0.0 0.0 112648 960 pts/3 S+ 04:39 0:00 grep --color=auto ambari

[root@ec2-xxxx ec2-user]# ps aux | grep ambari

root 16253 0.0 0.1 385276 17736 ? Sl 04:19 0:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start --expected-hostname=ec2-xxxx.ap-southeast-2.compute.amazonaws.com

root 16346 0.0 0.0 112648 956 pts/3 S+ 04:39 0:00 grep --color=auto ambari

[#12]

@David Sheard I am facing the same issue, please let me know in case you were able to resolve it.

[#13 · Accepted solution]

Ok sorted

yum remove ambari-agent on the servers where I did a manual install and rerun using Ambari

[#14]

@David Sheard @Saisubramaniam Gopalakrishnan

Public IPv4 addresses enable communication over the Internet, while private IPv4 addresses enable communication within the network of the instance (either EC2-Classic or a VPC)

Can you try using the private IP's in the /etc/hosts on all the hosts and retry.

1x2.31.83.454                   ip-1xx-x1-2x-1xx.ap.southeast-2.compute.internal       
1x2.91.27.784                   ip-2xx-x1-2x-1xx.ap.southeast-2.compute.internal 

Please let me know