question Re: Openssl error upon host registration in Archives of Support Questions (Read Only)

Openssl error upon host registration

rgarcia — Thu, 24 Sep 2015 03:02:13 GMT

I'm trying to register hostnames in Ambari but getting the error below. We tried to run yum update openssl but its got the latest version. We tried to run yum - update it didn’t help. I also tried removing the other openssl packages openssl-devel-1.0.1e-42.el6.x86_64, openssl098e-0.9.8e-17.el6.centos.2.x86_64 and restarting the ambari server/agents and same error. Tried running yum -y install opens* as well but didn’t help. Any ideas?

ERROR 2015-09-23 09:47:07,402 NetUtil.py:77 - [Errno 8] _ssl.c:492: EOF occurred in violation of protocol
ERROR 2015-09-23 09:47:07,402 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2015-09-23 09:47:07,402 NetUtil.py:105 - Server at https://test.org:8440is not reachable, sleeping for 10 seconds...
WARNING 2015-09-23 09:47:07,402 NetUtil.py:105 - Server at https://test.org:8440is not reachable, sleeping for 10 seconds...
INFO 2015-09-23 09:47:14,746 main.py:74 - loglevel=logging.INFO
INFO 2015-09-23 09:47:14,746 main.py:74 - loglevel=logging.INFO
INFO 2015-09-23 09:47:17,403 NetUtil.py:59 - Connecting to https://test.org:8440/ca
WARNING 2015-09-23 09:47:17,404 NetUtil.py:82 - Failed to connect to https://test.org:8440/ca due to [Errno 111] Connection refused
WARNING 2015-09-23 09:47:17,404 NetUtil.py:105 - Server at https://test.org:8440is not reachable, sleeping for 10 seconds...
WARNING 2015-09-23 09:47:17,404 NetUtil.py:105 - Server at https://test.org:8440is not reachable, sleeping for 10 seconds...
ERROR 2015-09-23 09:47:19,780 main.py:315 - Fatal exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 312, in <module>
    main(heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 248, in main
    stop_agent()
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 198, in stop_agent
    sys.exit(1)
SystemExit: 1
ERROR 2015-09-23 09:47:19,780 main.py:315 - Fatal exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 312, in <module>
    main(heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 248, in main
    stop_agent()
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 198, in stop_agent
    sys.exit(1)
SystemExit: 1

Re: Openssl error upon host registration

smohanty — Thu, 24 Sep 2015 03:20:26 GMT

What is the jdk being used?

We have noticed issues issues when open jdk 1.7.0 is used with Ambari Server resulted in the ssl issue?

Re: Openssl error upon host registration

rgarcia — Thu, 24 Sep 2015 03:22:09 GMT

I believe its using openjdk 1.7. How do I check this? There are several jdk versions installed on all machines.

Re: Openssl error upon host registration

smohanty — Thu, 24 Sep 2015 03:23:53 GMT

Check for the value in ambari.properties file:

java.home=/usr/jdk64/jdk1.8.0_40

jdk.name=jdk-8u40-linux-x64.tar.gz

If there are multiple jdks installed try switching the jdk.

ambari-server setup –j <jdk path>

http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_Installing_HDP_AMB/content/_setup_options.html

Re: Openssl error upon host registration

rgarcia — Thu, 24 Sep 2015 03:24:37 GMT

Got it. It’s using OpenJDK 1.7.0_85. Is this version an issue with HDP 2.3?

Re: Openssl error upon host registration

smohanty — Thu, 24 Sep 2015 03:26:36 GMT

Yes. That is a known issue. Modify Ambari Server's java.home to the standard oracle jdk.

Re: Openssl error upon host registration

rgarcia — Thu, 24 Sep 2015 03:27:14 GMT

that fixed it. Thanks.

Re: Openssl error upon host registration

gaurav_sharma — Thu, 24 Dec 2015 21:21:23 GMT

Hi,

We are also facing same issue. Even after setting java.home=/usr/jdk64/jdk1.8.0_40 in ambari.properties , it is not resolved.

Is standard oracle jdk path different from this ?

Below are logs snippet from ambari-agent.log ::

ERROR 2015-12-24 18:43:31,604 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) ERROR 2015-12-24 18:43:31,604 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

Re: Openssl error upon host registration

ali_gouta — Fri, 25 Dec 2015 06:11:12 GMT

@gaurav sharma I am actually facing the same issue, I have the same SSL error message during the registration. Did you find a solution ?

Re: Openssl error upon host registration

gaurav_sharma — Sat, 26 Dec 2015 13:51:25 GMT

@Ali Gouta No, we are still trying to figure out the solution.

OpenSSl version in my case is ::

[root@dehdteste7778b54apse01 ~]# rpm -qa | grep openssl

openssl-devel-1.0.1k-13.88.amzn1.x86_64

openssl-1.0.1k-13.88.amzn1.x86_64

Can this be a problem ?

Re: Openssl error upon host registration

geko — Sun, 27 Dec 2015 04:25:46 GMT

Hi @Ali Gouta , @gaurav sharma ,

which one is your default Java, that is being used ?

alternatives --display java

ls -al /etc/alternatives/java

Can you ensure that you are using Oracle JDK?

afaik it is definitely something related to the Java Kit...

Re: Openssl error upon host registration

ali_gouta — Sun, 27 Dec 2015 07:04:20 GMT

ls -al /etc/alternatives/java :
/etc/alternatives/java -> /opt/jdk1.8.0_66/bin/java

nano /etc/ambari-server/conf/ambari.properties :
java.home=/usr/jdk64/jdk1.8.0_40
server.jdbc.postgres.schema=ambari
jdk.name=jdk-8u40-linux-x64.tar.gz

@Gerd Koenig To sum up my issue:

I was trying to install Kerberos using the Active directory KDC option, then I had an issue and I was told that I should add the ambari server to the list of the cluster hosts (i.e. Having an ambari agent running with the ambari server on the same host). My 5 nodes are running well, now in the Add Host Wizard, I want to register the machine hosting the ambari-server, however the SSL_Error pops up each time I try to make it. In the ambari-agent.log, I can see the following message:

WARNING 2015-12-26 23:55:33,924 NetUtil.py:105 - Server at https://manager.cluster.mediatvcom:8440 is not reachable, sleeping for 10 seconds...
WARNING 2015-12-26 23:55:33,924 NetUtil.py:105 - Server at https://manager.cluster.mediatvcom:8440 is not reachable, sleeping for 10 seconds...
INFO 2015-12-26 23:55:43,925 NetUtil.py:59 - Connecting to https://manager.cluster.mediatvcom:8440
ERROR 2015-12-26 23:55:44,099 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
ERROR 2015-12-26 23:55:44,099 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://manager.cluster.mediatvcom:8440 for more details.

For more details, I opened a thread 3 days ago regarding this issue:

https://community.hortonworks.com/questions/7886/g...

Thank you.

Re: Openssl error upon host registration

ali_gouta — Sun, 27 Dec 2015 07:11:33 GMT

@gaurav sharma

In the documentation, it is said that we should user OpenSSL (v1.01, build 16 or later). So I think you should upgrade your OpenSSL version. Actually, I am running openssl-1.0.1e-42.el6_7.1.x86_64, and still not being able to register my agent 😞

Re: Openssl error upon host registration

aervits — Sun, 27 Dec 2015 12:09:11 GMT

@Ali Gouta you're showing two different versions of Java. Notice one is 66 and the one in ambari.properties is 40. Modify the ambari.properties as @smohanty suggested to the latest version.

Re: Openssl error upon host registration

geko — Sun, 27 Dec 2015 19:02:50 GMT

Hi @Ali Gouta , I cross-checked your post you was referring to. In that post you mention you are(were) running Oracle JDK 1.7 and now you upgraded just the Ambari-node to JDK 1.8. Does this mean all the other cluster nodes are still running OracleJDK1.7 ?

And your output of the java version and property entry show two different versions.

I'd recommend to ensure that you are running the same ORACLE JDK on the ambari node as on the other cluster nodes. Compare by e.g.

#>java -version

#>alternatives --display java

#>rpm -qa | grep java

on Ambari node vs. another cluster node.

I got rid of that error message (some weeks back having the same issue, but with HDP2.2 version) by using ORACLE JDK1.7 throughout all the nodes. And don't forget to run

#>ambari-server setup -j <your-java-home>
e.g. if you have your java installed in /usr/jdk64/oraclejdk1.7/bin/java, then
#>ambari-server setup -j /usr/jdk64/oraclejdk1.7

And I totally agree to @Artem Ervits , at least fix your version mismatch on the ambari node itself. Therefore if you want to stick with your 1.8.0_66 version, then

#>ambari-server setup -j /opt/jdk1.8.0_66/

HTH, best regards...

Re: Openssl error upon host registration

ali_gouta — Mon, 28 Dec 2015 04:33:11 GMT

@Artem Ervits @Gerd Koenig

Thank you so much for your help. I followed your recommendations and I finally figured out what was messing up !! The main issue was related to python and not to jdk !!

Actually, all my agents were running python 2.6:

>> ambari-agent restart
Restarting ambari-agent
Verifying Python version compatibility...
Using python  /usr/bin/python2.6

Except the agent running on the ambari-server it was running python 2.7 for an unknow reason ! (I may have accidentally installed it before on that node ). A known workaround is to inject PYTHON=/usr/bin/python2.6 into /var/lib/ambari-agent/ambari-env.sh.

This problem is solved :).

Re: Openssl error upon host registration

gaurav_sharma — Mon, 28 Dec 2015 16:07:52 GMT

Hi @Gerd Koenig / @Artem Ervits

As suggested i tried ambari-setup with -j option . Below are details ::

[root@dehdteste7778b54apse01 ~]# ls -l /etc/alternatives/java

lrwxrwxrwx 1 root root 26 Dec 28 13:20 /etc/alternatives/java -> /usr/lib/jvm/java/bin/java

[root@dehdteste7778b54apse01 ~]# alternatives --display java

java - status is auto.

link currently points to /usr/lib/jvm/java/bin/java

/usr/lib/jvm/java/bin/java - priority 1062

Current `best' version is /usr/lib/jvm/java/bin/java.

[root@dehdteste7778b54apse01 ~]# java -version

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

[root@dehdteste7778b54apse01 ~]# cat /etc/ambari-server/conf/ambari.properties | grep -i java.home java.home=/usr/lib/jvm/java

But still i same error is there.

In ambari-server.log , i can see ::

WARN [qtp-ambari-agent-69] nio:720 - javax.net.ssl.SSLException: Received fatal alert: unknown_ca

In ambari-agent.log ::

INFO 2015-12-28 13:37:52,395 NetUtil.py:59 - Connecting to https://dehdteste7778b54apse01.in.domain.com:8... ERROR 2015-12-28 13:37:52,460 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) ERROR 2015-12-28 13:37:52,460 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.

Re: Openssl error upon host registration

darkz1984 — Thu, 24 Mar 2016 11:19:06 GMT

I am still not very clearly what's the resolve?

rpm -qa | grep java

tzdata-java-2014g-1.el6.noarch

java-1.6.0-openjdk-1.6.0.0-11.1.13.4.el6.x86_64

java-1.7.0-openjdk-1.7.0.95-2.6.4.0.el6_7.x86_64

alternatives --display java

java - status is auto. Current `best' version is /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java.

and the python problem @Ali Gouta said ,it still the same ....

Re: Openssl error upon host registration

darkz1984 — Tue, 29 Mar 2016 10:39:39 GMT

I use openjdk 1.8,then the problem is resolved.

So, I think is just the java version problem...

Re: Openssl error upon host registration

neil_hauge — Fri, 12 Aug 2016 07:59:39 GMT

This worked for me. Was using openJDK v1.8. Tried a bunch of things, but the only thing that worked was installing the Oracle JDK on Ambari and all nodes. You may have to use the "alternative --config java" to activate the correct JDK even when installing via RPM.