Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

Contributor

Dear experts,

I have installed HDP 2.4.3 on AWS EC2 instances and i am facing a problem that, the ambari-agent and server both are running fine but agent is not able to make connections to server. I have tried all possibilities suggested in the HDP forums but nothing worked. Could you please help me on this ? Below are some details

Versions

------------------

Red Hat Enterprise Linux Server release 7.3 (Maipo) on AWS EC2

Ambari 2.4.3.0

HDP 2.4.3

Python 2.7.5 (default, May 3 2017, 07:55:04)

ambari-agent service running

---------------------------

root 3313 1 0 01:22 pts/0 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start

root 3321 3313 0 01:22 pts/0 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start

ambari-env.sh file

-----------------------

AMBARI_PASSPHRASE="DEV"

export PATH=$PATH:/var/lib/ambari-agent

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.6/site-packages

ambari-agent.ini

------------------------

[security] keysdir=/var/lib/ambari-agent/keys

server_crt=ca.crt

passphrase_env_var_name=AMBARI_PASSPHRASE

ssl_verify_cert=0

JDK versions in ambari-server.properties

----------------------------------

java.home=/usr/jdk64/jdk1.8.0_77

java.releases=jdk1.8,jdk1.7

jdk1.7.desc=Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7

jdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

ambari-agent log

--------------------------------

INFO 2017-08-05 01:14:38,849 HeartbeatHandlers.py:115 - Stop event received INFO 2017-08-05 01:14:38,849 NetUtil.py:125 - Stop event received INFO 2017-08-05 01:14:38,849 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2017-08-05 01:14:38,850 ExitHelper.py:67 - Cleanup finished, exiting with code:0 INFO 2017-08-05 01:14:39,504 main.py:223 - Agent died gracefully, exiting. INFO 2017-08-05 01:14:39,505 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,820 DataCleaner.py:39 - Data cleanup thread started INFO 2017-08-05 01:18:09,822 DataCleaner.py:120 - Data cleanup started INFO 2017-08-05 01:18:09,826 DataCleaner.py:122 - Data cleanup finished INFO 2017-08-05 01:18:09,853 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2017-08-05 01:18:09,856 main.py:349 - Connecting to Ambari server at https://XXXXXXX:8440 (XXXXXX) INFO 2017-08-05 01:18:09,856 NetUtil.py:65 - Connecting to https://XXXXXXXXXXXXXXXXXXXXXXXX:8440/ca ERROR 2017-08-05 01:18:09,918 NetUtil.py:91 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) ERROR 2017-08-05 01:18:09,919 NetUtil.py:92 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. WARNING 2017-08-05 01:18:09,921 NetUtil.py:119 - Server at https://XXXXXXXXXXXXXXX:8440 is not reachable, sleeping for 10 seconds...

1 ACCEPTED SOLUTION

Super Mentor

@Chiranjeevi Nimmala


As you using Python version "python-2.7.5" or higher, hence you should try to either downgrade the python version to lower than python-2.7.5 as it causes this issue.

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)<br>

(OR)
Else you will need to following the steps mentioned in the following doc to fix the "certificate verify failed (_ssl.c" issue while using RHEL7: Controlling and troubleshooting certificate verification
https://access.redhat.com/articles/2039753#controlling-certificate-verification-7

.

Also we will suggest you to remove the Ambari Server certificates as mentioned in the following doc so that new certificates can be generated by ambari.

https://cwiki.apache.org/confluence/display/AMBARI/Handling+Expired+HTTPS+Certificates

https://community.hortonworks.com/articles/68799/steps-to-fix-ambari-server-agent-expired-certs.html

.

View solution in original post

14 REPLIES 14

Super Mentor

@Chiranjeevi Nimmala


As you using Python version "python-2.7.5" or higher, hence you should try to either downgrade the python version to lower than python-2.7.5 as it causes this issue.

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)<br>

(OR)
Else you will need to following the steps mentioned in the following doc to fix the "certificate verify failed (_ssl.c" issue while using RHEL7: Controlling and troubleshooting certificate verification
https://access.redhat.com/articles/2039753#controlling-certificate-verification-7

.

Also we will suggest you to remove the Ambari Server certificates as mentioned in the following doc so that new certificates can be generated by ambari.

https://cwiki.apache.org/confluence/display/AMBARI/Handling+Expired+HTTPS+Certificates

https://community.hortonworks.com/articles/68799/steps-to-fix-ambari-server-agent-expired-certs.html

.

Contributor

Thank you, disabling the certificates as mentioned in https://access.redhat.com/articles/2039753#controlling-certificate-verification-7 helped

Rising Star

@Chiranjeevi Nimmala

What change you made in order to resolve this issue? Should I change to verify=enable or do I need to make some changes in Ambari server/agent confs?

Thank you.

Rising Star

NM, I resolved this by downgrading python version from 2.7.5-58 to 2.7.5-48. Earlier we were running 2.7.5-48, OS patching upgraded python version to 2.7.5-58 which started throwing SSL exception.

Hope this helps someone in future.

Explorer

Can you give me the steps to downgrade python 2.7.5-58 to 2.7.5-48 as I am getting SSL connection failed error??

Thanks to Jay SenSharma,

Performing following command on every agent node helped me:

sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg

Super Mentor

@Dima Kovalyov

Thank you for sharing the solution. This "sed" command will be quick and good to change on a large cluster.

Cloudera Employee

this one works for me, and don't need to regenerate key for ambari server and other ambari agents

New Contributor

Thank you very much

Its works good

New Contributor

ERROR 2018-10-03 19:31:19,461 ServerAPI.py:154 - POST https://hdp:9441/api/v1/unregister/hdp failed. (SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)
INFO 2018-10-03 19:31:19,471 security.py:178 - Server certificate not exists, downloading
INFO 2018-10-03 19:31:19,471 security.py:191 - Downloading server cert from https://hdp:9440/cert/ca/
ERROR 2018-10-03 19:31:19,625 ServerAPI.py:318 - GET https://hdp:9441/api/v1/schedule failed. (SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)

Was getting this error all over my agent. I have change below property in the /etc/python/cert-verification.cfg file -

$ cat /etc/python/cert-verification.cfg
# Possible values are:
# 'enable' to ensure HTTPS certificate verification is enabled by default
# 'disable' to ensure HTTPS certificate verification is disabled by default
# 'platform_default' to delegate the decision to the redistributor providing this particular Python version

# For more info refer to https://www.python.org/dev/peps/pep-0493/
[https]
verify=disable

Was able to resolve the issue.

New Contributor

Hi everyone!

I installed the ambari server and ambari-agent to my laptop (SuSE Tumbleweed 2019) to set up a local single-node Hadoop cluster. (Server and agent versions 2.1.2.1-418)

Server starts without issues, and I can open the console at http://localhost:8080

Agent says it started, but it can't connect the server, here's what the log says :

ERROR 2019-07-15 11:06:06,862 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)ERROR 2019-07-15 11:06:06,862 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.


I have python (2.7.16-2.1) installed. Also python3 is installed, but the executable defaults to version 2.7 :

ls -l /usr/bin/python
lrwxrwxrwx 1 root root 9 Jun 24 22:12 /usr/bin/python -> python2.7


I would implement this "verify=disable" solution mentioned above, but there is no file named cert-verification.cfg in my system. There is no python dir under etc at all !

So I think maybe python under SuSE might have a different configuration than other distributions. I google'd a lot, but couldn't find a solution for this yet.


Can someone guide me thru this issue please ?




New Contributor

Hi everyone!

I installed the ambari server and ambari-agent to my laptop (SuSE Tumbleweed 2019) to set up a local single-node Hadoop cluster. (Server and agent versions 2.1.2.1-418)

Server starts without issues, and I can open the console at http://localhost:8080

Agent says it started, but it can't connect the server, here's what the log says :

  1. ERROR 2019-07-15 11:06:06,862 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)ERROR 2019-07-15 11:06:06,862 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.


I have python (2.7.16-2.1) installed. Also python3 is installed, but the executable defaults to version 2.7 :

  1. ls -l /usr/bin/python
  2. lrwxrwxrwx 1 root root 9 Jun 24 22:12 /usr/bin/python -> python2.7


I would implement this "verify=disable" solution mentioned above, but there is no file named cert-verification.cfg in my system. There is no python dir under etc at all !

So I think maybe python under SuSE might have a different configuration than other distributions. I google'd a lot, but couldn't find a solution for this yet.


Can someone guide me thru this issue please ?