question Re: OpenSSL compatibility in Support Questions

OpenSSL compatibility

darko_milovanov — Tue, 15 Aug 2017 23:49:04 GMT

We recently patched Linux servers (RHEL 7.4). As a part of the patching we upgraded openSSL libraries (from 1.0.1e-60.el7_3.1.x86_64 to 1.0.2k-8.el7.x86_64). After completing the process, we saw Heartbeat Lost message in Ambari UI. When I tried to run ambari-agent restart command and got this message in log file:

INFO 2017-08-13 09:04:31,873 NetUtil.py:62 - Connecting to https://servername.com:8440/ca ERROR 2017-08-13 09:04:31,942 NetUtil.py:88 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) ERROR 2017-08-13 09:04:31,942 NetUtil.py:89 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. WARNING 2017-08-13 09:04:31,943 NetUtil.py:116 - Server at https://servername.com:8440 is not reachable, sleeping for 10 seconds...

Prior to patching, we had everything configured properly. We are using Apache Ambari 2.4.2.0.

Is there any compatibility issue with OpenSSL and Ambari?

Thanks,
Darko

Re: OpenSSL compatibility

jsensharma — Tue, 15 Aug 2017 23:52:06 GMT

@Darko Milovanovic

Your issue looks similar to : https://community.hortonworks.com/questions/120861/ambari-agent-ssl-certificate-verify-failed-certifi.html

So please check if you using Python version "python-2.7.5" or higher, if yes then you should try to either downgrade the python version to lower than python-2.7.5 as it causes this issue.

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)

(OR)

Else you will need to following the steps mentioned in the following doc to fix the "certificate verify failed (_ssl.c" issue while using RHEL7: Controlling and troubleshooting certificate verification
https://access.redhat.com/articles/2039753#controlling-certificate-verification-7

Re: OpenSSL compatibility

darko_milovanov — Wed, 16 Aug 2017 20:23:48 GMT

Thank you for quick response! That was an issue and we were able to fix it.

Re: OpenSSL compatibility

jsensharma — Wed, 16 Aug 2017 20:26:45 GMT

@Darko Milovanovic

Good to know that the link was useful to you and it resolved your issue. It will be great if you can mark this thread as "Accepted" (Answered) that way it will be very easy for other HCC users to quickly browse the answered threads.

Re: OpenSSL compatibility

nnat25191 — Wed, 20 Sep 2017 21:18:14 GMT

THank you so much @Jay SenSharma. The link solved my problem. I'm posting it here in hoping it could help someone since I've been stuck with this for a day.

modify all nodes

vi /etc/python/cert-verification.cfg
[https]
verify=disable

Re: OpenSSL compatibility

sree2 — Fri, 04 May 2018 00:37:06 GMT

I have tried the following, but issue not resolved yet. Any other ideas will be helpful.

vi /etc/python/cert-verification.cfg
[https]
verify=disable

==========================

Running setup agent script...

==========================

Command start time 2018-05-03 12:22:38

("INFO 2018-05-03 12:17:42,290 HeartbeatHandlers.py:116 - Stop event received

INFO 2018-05-03 12:17:42,290 NetUtil.py:130 - Stop event received

INFO 2018-05-03 12:17:42,290 ExitHelper.py:56 - Performing cleanup before exiting...

INFO 2018-05-03 12:17:42,290 ExitHelper.py:70 - Cleanup finished, exiting with code:0

INFO 2018-05-03 12:17:42,337 main.py:283 - Agent died gracefully, exiting.

INFO 2018-05-03 12:17:42,338 ExitHelper.py:56 - Performing cleanup before exiting...

INFO 2018-05-03 12:22:40,287 main.py:145 - loglevel=logging.INFO

INFO 2018-05-03 12:22:40,288 main.py:145 - loglevel=logging.INFO

INFO 2018-05-03 12:22:40,288 main.py:145 - loglevel=logging.INFO

INFO 2018-05-03 12:22:40,289 DataCleaner.py:39 - Data cleanup thread started

INFO 2018-05-03 12:22:40,291 DataCleaner.py:120 - Data cleanup started

INFO 2018-05-03 12:22:40,291 DataCleaner.py:122 - Data cleanup finished

INFO 2018-05-03 12:22:40,291 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'ip-172-31-71-94.ec2.internal' using socket.getfqdn().

INFO 2018-05-03 12:22:40,297 PingPortListener.py:50 - Ping port listener started on port: 8670

INFO 2018-05-03 12:22:40,299 main.py:437 - Connecting to Ambari server at https://ip-172-31-xx-94.ec2.internal:8440 (172.31.71.94)

INFO 2018-05-03 12:22:40,299 NetUtil.py:70 - Connecting to https://ip-172-31-xx-94.ec2.internal:8440/ca

ERROR 2018-05-03 12:22:40,313 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-05-03 12:22:40,313 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.

Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-05-03 12:22:40,313 NetUtil.py:124 - Server at https://ip-172-31-xx-94.ec2.internal:8440 is not reachable, sleeping for 10 seconds...

", None)

("INFO 2018-05-03 12:17:42,290 HeartbeatHandlers.py:116 - Stop event received

INFO 2018-05-03 12:17:42,290 NetUtil.py:130 - Stop event received

INFO 2018-05-03 12:17:42,290 ExitHelper.py:56 - Performing cleanup before exiting...

INFO 2018-05-03 12:17:42,290 ExitHelper.py:70 - Cleanup finished, exiting with code:0

INFO 2018-05-03 12:17:42,337 main.py:283 - Agent died gracefully, exiting.

INFO 2018-05-03 12:17:42,338 ExitHelper.py:56 - Performing cleanup before exiting...

INFO 2018-05-03 12:22:40,287 main.py:145 - loglevel=logging.INFO

INFO 2018-05-03 12:22:40,288 main.py:145 - loglevel=logging.INFO

INFO 2018-05-03 12:22:40,288 main.py:145 - loglevel=logging.INFO

INFO 2018-05-03 12:22:40,289 DataCleaner.py:39 - Data cleanup thread started

INFO 2018-05-03 12:22:40,291 DataCleaner.py:120 - Data cleanup started

INFO 2018-05-03 12:22:40,291 DataCleaner.py:122 - Data cleanup finished

INFO 2018-05-03 12:22:40,291 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'ip-172-31-71-94.ec2.internal' using socket.getfqdn().

INFO 2018-05-03 12:22:40,297 PingPortListener.py:50 - Ping port listener started on port: 8670

INFO 2018-05-03 12:22:40,299 main.py:437 - Connecting to Ambari server at https://ip-172-31-xx-94.ec2.internal:8440 (172.31.xx.94)

INFO 2018-05-03 12:22:40,299 NetUtil.py:70 - Connecting to https://ip-172-31-xx-94.ec2.internal:8440/ca

ERROR 2018-05-03 12:22:40,313 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-05-03 12:22:40,313 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.

Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-05-03 12:22:40,313 NetUtil.py:124 - Server at https://ip-172-31-xx-94.ec2.internal:8440 is not reachable, sleeping for 10 seconds...

", None)

Connection to ip-172-31-71-94.ec2.internal closed.

SSH command execution finished

host=ip-172-31-71-94.ec2.internal, exitcode=0

Command end time 2018-05-03 12:22:43

Registering with the server...

Registration with the server failed.

Re: OpenSSL compatibility

binglvabc — Tue, 08 May 2018 15:12:07 GMT

We have over come the problem by adding

following option to security section in ambari-agent.ini in all the hosts in the cluster:

[security] force_https_protocol=PROTOCOL_TLSv1_2

Re: OpenSSL compatibility

devenfan — Tue, 05 Jun 2018 23:28:59 GMT

Thanks to @bing lv , we have over come this issue by adding below config in [security] section of /etc/ambari-agent/conf/ambari-agent.ini

force_https_protocol=PROTOCOL_TLSv1_2

Re: OpenSSL compatibility

binglvabc — Wed, 04 Jul 2018 17:22:42 GMT

You are welcome

Re: OpenSSL compatibility

prashant_band — Tue, 17 Jul 2018 13:53:54 GMT

[ RESOLVED ]

Gone through same issue only when we are using oVirt Virtualization For our cluster deployment.

Only following solution resolved the problem (Thanks to @bing lv and @Deven Fan:

By adding below config in [security] section of

vi /etc/ambari-agent/conf/ambari-agent.ini
force_https_protocol=PROTOCOL_TLSv1_2

vi /etc/python/cert-verification.cfg 
[https] 
verify=disable

Re: OpenSSL compatibility

borayuret — Thu, 19 Jul 2018 20:40:33 GMT

Thanks. Solved my problem.

Re: OpenSSL compatibility

l_leszewski — Sat, 28 Jul 2018 21:38:02 GMT

I have the same issues on AWS servers. I'm going through ambari wizard and I always get failed status. In error as usual:

ERROR 2018-07-28 14:12:35,131 NetUtil.py:88 - EOF occurred in violation of protocol (_ssl.c:579) ERROR 2018-07-28 14:12:35,131 NetUtil.py:89 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. WARNING 2018-07-28 14:12:35,132 NetUtil.py:116 - Server at https://ip-172-31-0-xx.eu-west-1.compute.internal:8440 is not reachable, sleeping for 10 seconds... ', None) ('WARNING 2018-07-28 14:12:32,307 NetUtil.py:116 - Server at https://ip-172-31-0-xx.eu-west-1.compute.internal:8440 is not reachable, sleeping for 10 seconds... INFO 2018-07-28 14:12:32,307 HeartbeatHandlers.py:115 - Stop event received

I've tried adding in /etc/python/cert-verification.cfg

[https]
verify=disable

	I've tried adding in /etc/amabri-agent/conf/ambari-agent.in
[security]
force_https_protocol=PROTOCOL_TLSv1_2

	I've restarted agents still the same error 😞 Any ideas? 🙂

Re: OpenSSL compatibility

bsaoula — Mon, 30 Jul 2018 22:19:31 GMT

Hello

I've just add these two line below under security section and it works
[security]
ssl_verify_cert=0
force_https_protocol=PROTOCOL_TLSv1_2

Re: OpenSSL compatibility

l_leszewski — Tue, 31 Jul 2018 20:58:53 GMT

Ok for future users 🙂

Check if certyficate is generated by ambari server from one of the nodes:

openssl s_client -connect server_address:8440

corect results (similar):

---Server certificate-----BEGIN CERTIFICATE-----

MIIFnDCCA4SgAwIBAgIBATANBgkqhkiG9w0BAQsFADBCMQswCQYDVQQGEwJYWDEV
.................
.................

If you are not receiving corect handshake you need to verify ambari-server.ini (ambari server):

vi /etc/ambari-server/conf/ambari.properties

and # the line with TLS cyphers 🙂

Ok for future users 🙂

Check if certyficate is generated by ambari server from one of the nodes:

openssl s_client -connect server_address:8440

corect results (similar):

---Server certificate-----BEGIN CERTIFICATE-----

MIIFnDCCA4SgAwIBAgIBATANBgkqhkiG9w0BAQsFADBCMQswCQYDVQQGEwJYWDEV
.................
.................

If you are not receiving corect handshake you need to verify ambari-server.ini (ambari server):

vi /etc/ambari-server/conf/ambari.properties

and # the line with TLS cyphers 🙂