Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

[#1]

Dear experts,

I have installed HDP 2.4.3 on AWS EC2 instances and i am facing a problem that, the ambari-agent and server both are running fine but agent is not able to make connections to server. I have tried all possibilities suggested in the HDP forums but nothing worked. Could you please help me on this ? Below are some details

Versions

------------------

Red Hat Enterprise Linux Server release 7.3 (Maipo) on AWS EC2

Ambari 2.4.3.0

HDP 2.4.3

Python 2.7.5 (default, May 3 2017, 07:55:04)

ambari-agent service running

---------------------------

root 3313 1 0 01:22 pts/0 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start

root 3321 3313 0 01:22 pts/0 00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start

ambari-env.sh file

-----------------------

AMBARI_PASSPHRASE="DEV"

export PATH=$PATH:/var/lib/ambari-agent

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.6/site-packages

ambari-agent.ini

------------------------

[security] keysdir=/var/lib/ambari-agent/keys

server_crt=ca.crt

passphrase_env_var_name=AMBARI_PASSPHRASE

ssl_verify_cert=0

JDK versions in ambari-server.properties

----------------------------------

java.home=/usr/jdk64/jdk1.8.0_77

java.releases=jdk1.8,jdk1.7

jdk1.7.desc=Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7

jdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

ambari-agent log

--------------------------------

INFO 2017-08-05 01:14:38,849 HeartbeatHandlers.py:115 - Stop event received INFO 2017-08-05 01:14:38,849 NetUtil.py:125 - Stop event received INFO 2017-08-05 01:14:38,849 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2017-08-05 01:14:38,850 ExitHelper.py:67 - Cleanup finished, exiting with code:0 INFO 2017-08-05 01:14:39,504 main.py:223 - Agent died gracefully, exiting. INFO 2017-08-05 01:14:39,505 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,819 main.py:90 - loglevel=logging.INFO INFO 2017-08-05 01:18:09,820 DataCleaner.py:39 - Data cleanup thread started INFO 2017-08-05 01:18:09,822 DataCleaner.py:120 - Data cleanup started INFO 2017-08-05 01:18:09,826 DataCleaner.py:122 - Data cleanup finished INFO 2017-08-05 01:18:09,853 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2017-08-05 01:18:09,856 main.py:349 - Connecting to Ambari server at https://XXXXXXX:8440 (XXXXXX) INFO 2017-08-05 01:18:09,856 NetUtil.py:65 - Connecting to https://XXXXXXXXXXXXXXXXXXXXXXXX:8440/ca ERROR 2017-08-05 01:18:09,918 NetUtil.py:91 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) ERROR 2017-08-05 01:18:09,919 NetUtil.py:92 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. WARNING 2017-08-05 01:18:09,921 NetUtil.py:119 - Server at https://XXXXXXXXXXXXXXX:8440 is not reachable, sleeping for 10 seconds...

[#2]

Its works good

[#3]

For more details, please refer to this article: https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.h...

[#4]

ERROR 2018-10-03 19:31:19,461 ServerAPI.py:154 - POST https://hdp:9441/api/v1/unregister/hdp failed. (SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)
INFO 2018-10-03 19:31:19,471 security.py:178 - Server certificate not exists, downloading
INFO 2018-10-03 19:31:19,471 security.py:191 - Downloading server cert from https://hdp:9440/cert/ca/
ERROR 2018-10-03 19:31:19,625 ServerAPI.py:318 - GET https://hdp:9441/api/v1/schedule failed. (SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)

Was getting this error all over my agent. I have change below property in the /etc/python/cert-verification.cfg file -

$ cat /etc/python/cert-verification.cfg
# Possible values are:
# 'enable' to ensure HTTPS certificate verification is enabled by default
# 'disable' to ensure HTTPS certificate verification is disabled by default
# 'platform_default' to delegate the decision to the redistributor providing this particular Python version

# For more info refer to https://www.python.org/dev/peps/pep-0493/
[https]
verify=disable

Was able to resolve the issue.

[#5]

Hi everyone!

I installed the ambari server and ambari-agent to my laptop (SuSE Tumbleweed 2019) to set up a local single-node Hadoop cluster. (Server and agent versions 2.1.2.1-418)

Server starts without issues, and I can open the console at http://localhost:8080

Agent says it started, but it can't connect the server, here's what the log says :

ERROR 2019-07-15 11:06:06,862 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)ERROR 2019-07-15 11:06:06,862 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

I have python (2.7.16-2.1) installed. Also python3 is installed, but the executable defaults to version 2.7 :

ls -l /usr/bin/python
lrwxrwxrwx 1 root root 9 Jun 24 22:12 /usr/bin/python -> python2.7

I would implement this "verify=disable" solution mentioned above, but there is no file named cert-verification.cfg in my system. There is no python dir under etc at all !

So I think maybe python under SuSE might have a different configuration than other distributions. I google'd a lot, but couldn't find a solution for this yet.


Can someone guide me thru this issue please ?

[#6]

Hi everyone!

I installed the ambari server and ambari-agent to my laptop (SuSE Tumbleweed 2019) to set up a local single-node Hadoop cluster. (Server and agent versions 2.1.2.1-418)

Server starts without issues, and I can open the console at http://localhost:8080

Agent says it started, but it can't connect the server, here's what the log says :

ERROR 2019-07-15 11:06:06,862 NetUtil.py:77 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)ERROR 2019-07-15 11:06:06,862 NetUtil.py:78 - SSLError: Failed to connect. Please check openssl library versions.Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

I have python (2.7.16-2.1) installed. Also python3 is installed, but the executable defaults to version 2.7 :

ls -l /usr/bin/python
lrwxrwxrwx 1 root root 9 Jun 24 22:12 /usr/bin/python -> python2.7

I would implement this "verify=disable" solution mentioned above, but there is no file named cert-verification.cfg in my system. There is no python dir under etc at all !

So I think maybe python under SuSE might have a different configuration than other distributions. I google'd a lot, but couldn't find a solution for this yet.


Can someone guide me thru this issue please ?