Ambari fails to register on AWS

[#1]

I have created 3 AWS EC2 RHEL-7 instances, installed ambari server v2.2 (with jdk1.7) on 1 of them, then try to run the "create cluster" wizard. Then Ambari, even though passes all steps, fails in the final step to register agent with the server. The only node it succeeds to register is itself (the instance where ambari server is already running). It seems to me that there is an issue with the below call:

Connecting to https://ip-172-31-31-118.ec2.internal:8440/ca ', None

Security groups allow for all connections and no firewalls are running. Not completely sure if it can be SSL related or something else to do with java (I have already tried with jdk1.8). The ambari server node that succeeds generates the following log lines:

INFO 2016-01-19 19:19:36,639 NetUtil.py:60 - Connecting to https://ip-172-31-31-118.ec2.internal:8440/connection_info
INFO 2016-01-19 19:19:36,754 security.py:99 - SSL Connect being called.. connecting to the server
INFO 2016-01-19 19:19:36,917 security.py:60 - SSL connection established. Two-way SSL authentication is turned off on the server.

Please find below the whole error log from nodes that failed:
	==========================
Running setup agent script...
==========================
Command start time 2016-01-19 19:19:31
('INFO 2016-01-19 19:19:54,528 main.py:71 - loglevel=logging.INFO
INFO 2016-01-19 19:19:54,528 main.py:71 - loglevel=logging.INFO
INFO 2016-01-19 19:19:54,530 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-01-19 19:19:54,531 DataCleaner.py:120 - Data cleanup started
INFO 2016-01-19 19:19:54,531 DataCleaner.py:122 - Data cleanup finished
INFO 2016-01-19 19:19:54,538 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2016-01-19 19:19:54,540 main.py:289 - Connecting to Ambari server at https://ip-172-31-31-118.ec2.internal:8440 (172.31.31.118)
INFO 2016-01-19 19:19:54,540 NetUtil.py:60 - Connecting to https://ip-172-31-31-118.ec2.internal:8440/ca
', None)
('INFO 2016-01-19 19:19:54,528 main.py:71 - loglevel=logging.INFO
INFO 2016-01-19 19:19:54,528 main.py:71 - loglevel=logging.INFO
INFO 2016-01-19 19:19:54,530 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-01-19 19:19:54,531 DataCleaner.py:120 - Data cleanup started
INFO 2016-01-19 19:19:54,531 DataCleaner.py:122 - Data cleanup finished
INFO 2016-01-19 19:19:54,538 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2016-01-19 19:19:54,540 main.py:289 - Connecting to Ambari server at https://ip-172-31-31-118.ec2.internal:8440 (172.31.31.118)
INFO 2016-01-19 19:19:54,540 NetUtil.py:60 - Connecting to https://ip-172-31-31-118.ec2.internal:8440/ca
', None)
Connection to ip-172-31-20-141.ec2.internal closed.
SSH command execution finished
host=ip-172-31-20-141.ec2.internal, exitcode=0
Command end time 2016-01-19 19:19:56
Registering with the server...
Registration with the server failed.

[#2 · Accepted solution]

@Kyriakos Spyropoulos

Maybe your security group is preventing your node from connecting connect back to the Ambari Server.

Log onto one of the other nodes and try telnet back to the Ambari server on port 8670.

Here is an example of a simple configuration to allow the internal ports to be open to each other.

"You will obviously need to replace “[Your Public IP]/32” with your IP address and subnet mask you wish to access the cluster from. In this example the /32 denotes a single host that we will access the cluster from. Also replace “[Your Security Group]” with the id of your actual security group. You will notice that for brevity sake we have just opened all ports internally to the security group. They are not accessible to the outside world but only between the nodes in the cluster. In a production environment you would probably want to be very specific with regards to the ports opened internally as well."

See also this blog Deploying Hadoop Cluster Amazon ec2 Hortonworks

See also the answers to this question: Looking for Steps to Install HDP on AWS

[#3]

@Ancil McBarnett This is really good information. Thanks for sharing. Wikified 😉

[#4]

which version of jdk are you using, there were issues with certain versions of jdk? We had issue with openssl as well. You can also try to register ambari-agent manually on each node and test ssh between nodes, it has to work. @Kyriakos Spyropoulos

[#5]

So true... OpenSSL can be the issue also. See this link for more info.

https://community.hortonworks.com/questions/145/openssl-error-upon-host-registration.html

[#6]

Thank you all for prompt replies and extra thanks to @Ancil McBarnett your answer was toward the right direction. I enabled all traffic in my security group (for testing purposes) as opposed to only tcp traffic before. As seen in the above logs the right port for ambari-server is 8440. That resolved the issue and now agents can be registered through Ambari.

Just in sake of completeness my setup is:

OS: RHEL-7.2 Ambari: 2.1.2 JDK: 1.8 Python: 2.7