Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

[#1]

Dear experts,

I have installed HDP 2.4.3 on AWS EC2 instances and i am facing a problem that, the ambari-agent and server both are running fine but agent is not able to make connections to server. I have tried all possibilities suggested in the HDP forums but nothing worked. Could you please help me on this ? Below are some details

Versions

------------------

Red Hat Enterprise Linux Server release 7.3 (Maipo) on AWS EC2

Ambari 2.4.3.0

HDP 2.4.3

Python 2.7.5 (default, May 3 2017, 07:55:04)

ambari-agent service running

---------------------------

root 3313 1 0 01:22 pts/0 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start

root 3321 3313 0 01:22 pts/0 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start

ambari-env.sh file

-----------------------

AMBARI_PASSPHRASE="DEV"

export PATH=$PATH:/var/lib/ambari-agent

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.6/site-packages

ambari-agent.ini

------------------------

[security] keysdir=/var/lib/ambari-agent/keys

server_crt=ca.crt

passphrase_env_var_name=AMBARI_PASSPHRASE

ssl_verify_cert=0

JDK versions in ambari-server.properties

----------------------------------

java.home=/usr/jdk64/jdk1.8.0_77

java.releases=jdk1.8,jdk1.7

jdk1.7.desc=Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7

jdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

ambari-agent log

--------------------------------

INFO 2017-08-05 01:14:38,849 HeartbeatHandlers.py:115 - Stop event received INFO 2017-08-05 01:14:38,849 NetUtil.py:125 - Stop event received INFO 2017-08-05 01:14:38,849 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2017-08-05 01:14:38,850 ExitHelper.py:67 - Cleanup finished, exiting with code:0 INFO 2017-08-05 01:14:39,504 main.py:223 - Agent died gracefully, exiting. INFO 2017-08-05 01:14:39,505 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,820 DataCleaner.py:39 - Data cleanup thread started INFO 2017-08-05 01:18:09,822 DataCleaner.py:120 - Data cleanup started INFO 2017-08-05 01:18:09,826 DataCleaner.py:122 - Data cleanup finished INFO 2017-08-05 01:18:09,853 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2017-08-05 01:18:09,856 main.py:349 - Connecting to Ambari server at https://XXXXXXX:8440 (XXXXXX) INFO 2017-08-05 01:18:09,856 NetUtil.py:65 - Connecting to https://XXXXXXXXXXXXXXXXXXXXXXXX:8440/ca ERROR 2017-08-05 01:18:09,918 NetUtil.py:91 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) ERROR 2017-08-05 01:18:09,919 NetUtil.py:92 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. WARNING 2017-08-05 01:18:09,921 NetUtil.py:119 - Server at https://XXXXXXXXXXXXXXX:8440 is not reachable, sleeping for 10 seconds...

[#2 · Accepted solution]

@Chiranjeevi Nimmala

As you using Python version "python-2.7.5" or higher, hence you should try to either downgrade the python version to lower than python-2.7.5 as it causes this issue.

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)<br>

(OR)
Else you will need to following the steps mentioned in the following doc to fix the "certificate verify failed (_ssl.c" issue while using RHEL7: Controlling and troubleshooting certificate verification
https://access.redhat.com/articles/2039753#controlling-certificate-verification-7

.

Also we will suggest you to remove the Ambari Server certificates as mentioned in the following doc so that new certificates can be generated by ambari.

https://cwiki.apache.org/confluence/display/AMBARI/Handling+Expired+HTTPS+Certificates

https://community.hortonworks.com/articles/68799/steps-to-fix-ambari-server-agent-expired-certs.html

.

[#3]

Thank you, disabling the certificates as mentioned in https://access.redhat.com/articles/2039753#controlling-certificate-verification-7 helped

[#4]

@Chiranjeevi Nimmala

What change you made in order to resolve this issue? Should I change to verify=enable or do I need to make some changes in Ambari server/agent confs?

Thank you.

[#5]

NM, I resolved this by downgrading python version from 2.7.5-58 to 2.7.5-48. Earlier we were running 2.7.5-48, OS patching upgraded python version to 2.7.5-58 which started throwing SSL exception.

Hope this helps someone in future.

[#6]

Can you give me the steps to downgrade python 2.7.5-58 to 2.7.5-48 as I am getting SSL connection failed error??

[#7]

Thanks to Jay SenSharma,

Performing following command on every agent node helped me:

sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg

[#8]

@Dima Kovalyov

Thank you for sharing the solution. This "sed" command will be quick and good to change on a large cluster.

[#9]

this one works for me, and don't need to regenerate key for ambari server and other ambari agents

[#10]

Thank you very much