Openssl error upon host registration

[#1]

I'm trying to register hostnames in Ambari but getting the error below. We tried to run yum update openssl but its got the latest version. We tried to run yum - update it didn’t help. I also tried removing the other openssl packages openssl-devel-1.0.1e-42.el6.x86_64, openssl098e-0.9.8e-17.el6.centos.2.x86_64 and restarting the ambari server/agents and same error. Tried running yum -y install opens* as well but didn’t help. Any ideas?

ERROR 2015-09-23 09:47:07,402 NetUtil.py:77 - [Errno 8] _ssl.c:492: EOF occurred in violation of protocol
ERROR 2015-09-23 09:47:07,402 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2015-09-23 09:47:07,402 NetUtil.py:105 - Server at https://test.org:8440is not reachable, sleeping for 10 seconds...
WARNING 2015-09-23 09:47:07,402 NetUtil.py:105 - Server at https://test.org:8440is not reachable, sleeping for 10 seconds...
INFO 2015-09-23 09:47:14,746 main.py:74 - loglevel=logging.INFO
INFO 2015-09-23 09:47:14,746 main.py:74 - loglevel=logging.INFO
INFO 2015-09-23 09:47:17,403 NetUtil.py:59 - Connecting to https://test.org:8440/ca
WARNING 2015-09-23 09:47:17,404 NetUtil.py:82 - Failed to connect to https://test.org:8440/ca due to [Errno 111] Connection refused
WARNING 2015-09-23 09:47:17,404 NetUtil.py:105 - Server at https://test.org:8440is not reachable, sleeping for 10 seconds...
WARNING 2015-09-23 09:47:17,404 NetUtil.py:105 - Server at https://test.org:8440is not reachable, sleeping for 10 seconds...
ERROR 2015-09-23 09:47:19,780 main.py:315 - Fatal exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 312, in <module>
    main(heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 248, in main
    stop_agent()
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 198, in stop_agent
    sys.exit(1)
SystemExit: 1
ERROR 2015-09-23 09:47:19,780 main.py:315 - Fatal exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 312, in <module>
    main(heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 248, in main
    stop_agent()
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 198, in stop_agent
    sys.exit(1)
SystemExit: 1

[#2]

@gaurav sharma

In the documentation, it is said that we should user OpenSSL (v1.01, build 16 or later). So I think you should upgrade your OpenSSL version. Actually, I am running openssl-1.0.1e-42.el6_7.1.x86_64, and still not being able to register my agent 😞

[#3]

This worked for me. Was using openJDK v1.8. Tried a bunch of things, but the only thing that worked was installing the Oracle JDK on Ambari and all nodes. You may have to use the "alternative --config java" to activate the correct JDK even when installing via RPM.

[#4]

Hi @Ali Gouta , @gaurav sharma ,

which one is your default Java, that is being used ?

alternatives --display java

or

ls -al /etc/alternatives/java

Can you ensure that you are using Oracle JDK?

afaik it is definitely something related to the Java Kit...

[#5]

ls -al /etc/alternatives/java :
/etc/alternatives/java -> /opt/jdk1.8.0_66/bin/java

nano /etc/ambari-server/conf/ambari.properties :
java.home=/usr/jdk64/jdk1.8.0_40
server.jdbc.postgres.schema=ambari
jdk.name=jdk-8u40-linux-x64.tar.gz

@Gerd Koenig To sum up my issue:

I was trying to install Kerberos using the Active directory KDC option, then I had an issue and I was told that I should add the ambari server to the list of the cluster hosts (i.e. Having an ambari agent running with the ambari server on the same host). My 5 nodes are running well, now in the Add Host Wizard, I want to register the machine hosting the ambari-server, however the SSL_Error pops up each time I try to make it. In the ambari-agent.log, I can see the following message:

WARNING 2015-12-26 23:55:33,924 NetUtil.py:105 - Server at https://manager.cluster.mediatvcom:8440 is not reachable, sleeping for 10 seconds...
WARNING 2015-12-26 23:55:33,924 NetUtil.py:105 - Server at https://manager.cluster.mediatvcom:8440 is not reachable, sleeping for 10 seconds...
INFO 2015-12-26 23:55:43,925 NetUtil.py:59 - Connecting to https://manager.cluster.mediatvcom:8440
ERROR 2015-12-26 23:55:44,099 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
ERROR 2015-12-26 23:55:44,099 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://manager.cluster.mediatvcom:8440 for more details. 

For more details, I opened a thread 3 days ago regarding this issue:

https://community.hortonworks.com/questions/7886/g...

Thank you.

[#6]

@Ali Gouta you're showing two different versions of Java. Notice one is 66 and the one in ambari.properties is 40. Modify the ambari.properties as @smohanty suggested to the latest version.

[#7]

Hi @Ali Gouta , I cross-checked your post you was referring to. In that post you mention you are(were) running Oracle JDK 1.7 and now you upgraded just the Ambari-node to JDK 1.8. Does this mean all the other cluster nodes are still running OracleJDK1.7 ?

And your output of the java version and property entry show two different versions.

I'd recommend to ensure that you are running the same ORACLE JDK on the ambari node as on the other cluster nodes. Compare by e.g.

#>java -version

#>alternatives --display java

#>rpm -qa | grep java

on Ambari node vs. another cluster node.

I got rid of that error message (some weeks back having the same issue, but with HDP2.2 version) by using ORACLE JDK1.7 throughout all the nodes. And don't forget to run

#>ambari-server setup -j <your-java-home>
e.g. if you have your java installed in /usr/jdk64/oraclejdk1.7/bin/java, then
#>ambari-server setup -j /usr/jdk64/oraclejdk1.7

And I totally agree to @Artem Ervits , at least fix your version mismatch on the ambari node itself. Therefore if you want to stick with your 1.8.0_66 version, then

#>ambari-server setup -j /opt/jdk1.8.0_66/

HTH, best regards...

[#8]

@Artem Ervits @Gerd Koenig

Thank you so much for your help. I followed your recommendations and I finally figured out what was messing up !! The main issue was related to python and not to jdk !!

Actually, all my agents were running python 2.6:

>> ambari-agent restart
Restarting ambari-agent
Verifying Python version compatibility...
Using python  /usr/bin/python2.6

Except the agent running on the ambari-server it was running python 2.7 for an unknow reason ! (I may have accidentally installed it before on that node ). A known workaround is to inject PYTHON=/usr/bin/python2.6 into /var/lib/ambari-agent/ambari-env.sh.

This problem is solved :).

[#9]

Hi @Gerd Koenig / @Artem Ervits

As suggested i tried ambari-setup with -j option . Below are details ::

[root@dehdteste7778b54apse01 ~]# ls -l /etc/alternatives/java

lrwxrwxrwx 1 root root 26 Dec 28 13:20 /etc/alternatives/java -> /usr/lib/jvm/java/bin/java

[root@dehdteste7778b54apse01 ~]# alternatives --display java

java - status is auto.

link currently points to /usr/lib/jvm/java/bin/java

/usr/lib/jvm/java/bin/java - priority 1062

Current `best' version is /usr/lib/jvm/java/bin/java.

[root@dehdteste7778b54apse01 ~]# java -version

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

[root@dehdteste7778b54apse01 ~]# cat /etc/ambari-server/conf/ambari.properties | grep -i java.home java.home=/usr/lib/jvm/java

But still i same error is there.

In ambari-server.log , i can see ::

WARN [qtp-ambari-agent-69] nio:720 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca

In ambari-agent.log ::

INFO 2015-12-28 13:37:52,395 NetUtil.py:59 - Connecting to https://dehdteste7778b54apse01.in.domain.com:8... ERROR 2015-12-28 13:37:52,460 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) ERROR 2015-12-28 13:37:52,460 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.

[#10]

I am still not very clearly what's the resolve?

rpm -qa | grep java

tzdata-java-2014g-1.el6.noarch

java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64

java-1.7.0-openjdk-1.7.0.95-2.6.4.0.el6_7.x86_64

alternatives --display java

java - status is auto. Current `best' version is /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java.

and the python problem @Ali Gouta said ,it still the same ....

[#11]

I use openjdk 1.8,then the problem is resolved.

So, I think is just the java version problem...