question Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed in Support Questions

Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

chiru_tnk — Sat, 05 Aug 2017 12:59:16 GMT

Dear experts,

I have installed HDP 2.4.3 on AWS EC2 instances and i am facing a problem that, the ambari-agent and server both are running fine but agent is not able to make connections to server. I have tried all possibilities suggested in the HDP forums but nothing worked. Could you please help me on this ? Below are some details

Versions

------------------

Red Hat Enterprise Linux Server release 7.3 (Maipo) on AWS EC2

Ambari 2.4.3.0

HDP 2.4.3

Python 2.7.5 (default, May 3 2017, 07:55:04)

ambari-agent service running

---------------------------

root 3313 1 0 01:22 pts/0 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start

root 3321 3313 0 01:22 pts/0 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start

ambari-env.sh file

-----------------------

AMBARI_PASSPHRASE="DEV"

export PATH=$PATH:/var/lib/ambari-agent

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.6/site-packages

ambari-agent.ini

------------------------

[security] keysdir=/var/lib/ambari-agent/keys

server_crt=ca.crt

passphrase_env_var_name=AMBARI_PASSPHRASE

ssl_verify_cert=0

JDK versions in ambari-server.properties

----------------------------------

java.home=/usr/jdk64/jdk1.8.0_77

java.releases=jdk1.8,jdk1.7

jdk1.7.desc=Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7

jdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

ambari-agent log

--------------------------------

INFO 2017-08-05 01:14:38,849 HeartbeatHandlers.py:115 - Stop event received INFO 2017-08-05 01:14:38,849 NetUtil.py:125 - Stop event received INFO 2017-08-05 01:14:38,849 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2017-08-05 01:14:38,850 ExitHelper.py:67 - Cleanup finished, exiting with code:0 INFO 2017-08-05 01:14:39,504 main.py:223 - Agent died gracefully, exiting. INFO 2017-08-05 01:14:39,505 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,820 DataCleaner.py:39 - Data cleanup thread started INFO 2017-08-05 01:18:09,822 DataCleaner.py:120 - Data cleanup started INFO 2017-08-05 01:18:09,826 DataCleaner.py:122 - Data cleanup finished INFO 2017-08-05 01:18:09,853 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2017-08-05 01:18:09,856 main.py:349 - Connecting to Ambari server at https://XXXXXXX:8440 (XXXXXX) INFO 2017-08-05 01:18:09,856 NetUtil.py:65 - Connecting to https://XXXXXXXXXXXXXXXXXXXXXXXX:8440/ca ERROR 2017-08-05 01:18:09,918 NetUtil.py:91 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) ERROR 2017-08-05 01:18:09,919 NetUtil.py:92 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. WARNING 2017-08-05 01:18:09,921 NetUtil.py:119 - Server at https://XXXXXXXXXXXXXXX:8440 is not reachable, sleeping for 10 seconds...

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

jsensharma — Sat, 05 Aug 2017 13:44:51 GMT

@Chiranjeevi Nimmala

As you using Python version "python-2.7.5" or higher, hence you should try to either downgrade the python version to lower than python-2.7.5 as it causes this issue.

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)<br>

(OR)
Else you will need to following the steps mentioned in the following doc to fix the "certificate verify failed (_ssl.c" issue while using RHEL7: Controlling and troubleshooting certificate verification
https://access.redhat.com/articles/2039753#controlling-certificate-verification-7

Also we will suggest you to remove the Ambari Server certificates as mentioned in the following doc so that new certificates can be generated by ambari.

https://cwiki.apache.org/confluence/display/AMBARI/Handling+Expired+HTTPS+Certificates

https://community.hortonworks.com/articles/68799/steps-to-fix-ambari-server-agent-expired-certs.html

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

chiru_tnk — Sun, 06 Aug 2017 12:21:04 GMT

Thank you, disabling the certificates as mentioned in https://access.redhat.com/articles/2039753#controlling-certificate-verification-7 helped

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

shashant_panwar — Tue, 05 Sep 2017 23:16:48 GMT

@Chiranjeevi Nimmala

What change you made in order to resolve this issue? Should I change to verify=enable or do I need to make some changes in Ambari server/agent confs?

Thank you.

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

shashant_panwar — Wed, 06 Sep 2017 00:44:26 GMT

NM, I resolved this by downgrading python version from 2.7.5-58 to 2.7.5-48. Earlier we were running 2.7.5-48, OS patching upgraded python version to 2.7.5-58 which started throwing SSL exception.

Hope this helps someone in future.

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

Dima_Kovalyov — Wed, 27 Sep 2017 19:37:25 GMT

Thanks to Jay SenSharma,

Performing following command on every agent node helped me:

sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

jsensharma — Wed, 27 Sep 2017 19:59:08 GMT

@Dima Kovalyov

Thank you for sharing the solution. This "sed" command will be quick and good to change on a large cluster.

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

YangWang — Mon, 02 Oct 2017 20:07:17 GMT

this one works for me, and don't need to regenerate key for ambari server and other ambari agents

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

gauravb117 — Wed, 06 Dec 2017 14:03:01 GMT

Can you give me the steps to downgrade python 2.7.5-58 to 2.7.5-48 as I am getting SSL connection failed error??

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

mdtarikuzzaman — Thu, 18 Jan 2018 21:55:40 GMT

Its works good

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

pcodding — Wed, 16 May 2018 00:56:10 GMT

For more details, please refer to this article: https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.html

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

lisong1205 — Wed, 18 Jul 2018 09:03:50 GMT

Thank you very much

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

nishant_gupta59 — Thu, 04 Oct 2018 06:36:48 GMT

ERROR 2018-10-03 19:31:19,461 ServerAPI.py:154 - POST https://hdp:9441/api/v1/unregister/hdp failed. (SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)
INFO 2018-10-03 19:31:19,471 security.py:178 - Server certificate not exists, downloading
INFO 2018-10-03 19:31:19,471 security.py:191 - Downloading server cert from https://hdp:9440/cert/ca/
ERROR 2018-10-03 19:31:19,625 ServerAPI.py:318 - GET https://hdp:9441/api/v1/schedule failed. (SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)

Was getting this error all over my agent. I have change below property in the /etc/python/cert-verification.cfg file -

$ cat /etc/python/cert-verification.cfg
# Possible values are:
# 'enable' to ensure HTTPS certificate verification is enabled by default
# 'disable' to ensure HTTPS certificate verification is disabled by default
# 'platform_default' to delegate the decision to the redistributor providing this particular Python version

# For more info refer to https://www.python.org/dev/peps/pep-0493/
[https]
verify=disable

Was able to resolve the issue.

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

oguzeren — Mon, 15 Jul 2019 20:20:07 GMT

Hi everyone!

I installed the ambari server and ambari-agent to my laptop (SuSE Tumbleweed 2019) to set up a local single-node Hadoop cluster. (Server and agent versions 2.1.2.1-418)

Server starts without issues, and I can open the console at http://localhost:8080

Agent says it started, but it can't connect the server, here's what the log says :

ERROR 2019-07-15 11:06:06,862 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)ERROR 2019-07-15 11:06:06,862 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

I have python (2.7.16-2.1) installed. Also python3 is installed, but the executable defaults to version 2.7 :

ls -l /usr/bin/python
lrwxrwxrwx 1 root root 9 Jun 24 22:12 /usr/bin/python -> python2.7

I would implement this "verify=disable" solution mentioned above, but there is no file named cert-verification.cfg in my system. There is no python dir under etc at all !

So I think maybe python under SuSE might have a different configuration than other distributions. I google'd a lot, but couldn't find a solution for this yet.

Can someone guide me thru this issue please ?

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

oguzeren — Mon, 15 Jul 2019 20:20:18 GMT

ERROR 2019-07-15 11:06:06,862 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)ERROR 2019-07-15 11:06:06,862 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

I have python (2.7.16-2.1) installed. Also python3 is installed, but the executable defaults to version 2.7 :

ls -l /usr/bin/python
lrwxrwxrwx 1 root root 9 Jun 24 22:12 /usr/bin/python -> python2.7