Ambari-Server failing to communicate with Ambari-agent with the error CERTIFICATE_VERIFY_FAILED, Is it safe to disable the certificate verification in Python ?

[#1]

Ambari version: 2.2.2.18

HDP stack: 2.4.3

OS: centos 7.3

Issue description:

Ambari-server can't communicate with Ambari agent. I can see below error in the ambari-agent logs:

ERROR 2017-09-18 06:35:34,684 NetUtil.py:84 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) ERROR 2017-09-18 06:35:34,684 NetUtil.py:85 - SSLError: Failed to connect. Please check openssl library versions.

I am facing this issue recently and it appears this can be replicated consistently after the instances are restarted. (I am using EC2 instances).

I am able to register agent nodes successfully, install HDP cluster, run yarn jobs etc... no problem at all. Once i restart my instances, I see this problem.

There are some solutions already posted for this problem like:

• Downgrade the Python from 2.7 to lower. This is a known problem of Ambari with Python 2.7
• Control the certificate verification by disabling it.
  
  • Set "verify = disable"; under /etc/python/cert-verification.cfg

I don't want to play with Python as it can disrupt lot many things like Cassandra, yum package manager etc...

Second work around is very much easy and it works well!

Now comes my question :- Is it safe to disable the certificate verification in Python ? i.e. by setting property verify = disable

Regards,

Vinay MP

[#2]

@Vinay MP

Pleas echeck your python version . Is it "python-2.7.5" or higherif yes then should try to either downgrade the python version to lower than python-2.7.5 else it might causes this issue.

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)

(OR)

Else you will need to following the steps mentioned in the following doc to fix the "certificate verify failed (_ssl.c" issue while using RHEL7: Controlling and troubleshooting certificate verification

https://access.redhat.com/articles/2039753#controlling-certificate-verification-7 .

.

The second option is safe as long as you do not want the certificate verification (or if you are in test environment). Otherwise it is highly discouraged to disable it. The verify=disable, ensures that the HTTPS certificate verification is disabled. You can verify the same using the Sample code mentioned in the above link to see if it works at your end.

.

With Python 2.7.9 release the default HTTPS behaviour has changed, which is now to always verify the remote HTTPS certificate to which you are initiating a connection.

In the Python 2.7.9 release notes you can read more about the changes that made it in this release of Python, and PEP 476 provides the technical details and rationale about this change.

[#3]

@Vinay MP

The following Link, Describes why it was needed or why this feature was introduced: https://www.python.org/dev/peps/pep-0476/

.

CVE-2014-9365 The Python standard library HTTP client modules (such as httplib or urllib) did not perform verification of TLS/SSL certificates when connecting to HTTPS servers. A man-in-the-middle attacker could use this flaw to hijack connections and eavesdrop or modify transferred data.

[#4]

@Vinay MP

If this issue is resolved then can you please mark this thread as "Accepted". Or if you found any other solution then please share.

[#5]

@Jay SenSharma

Haven't found a feasible solution.

As mentioned in the issue description,

Downgrading Python 2.6 is not feasible as there are OS dependencies and based on below link:

https://stackoverflow.com/questions/46274499/ambari-agent-certificate-verify-failed-is-it-safe-to-di...

I have got a suggestion it's not a good idea to disable certificate verification in Python.

Sharing some more information from our investigation, Just thinking it might help others:

We use AWS EC2

With Python 2.7, JDK 1.8 and Cent OS 7.2 there is no issue. Everything is smooth.

With Python 2.7, JDK 1.8 and Cent OS 7.3 and Centos 7.4 we are seeing this issue.

What I have reported here, is with respect to Centos 7.3 and with Centos 7.4 Issue is slightly different:

Certificate verification fails while adding nodes to the cluster itself.

Downgrading from centos 7.3 to 7.2 is not straight forward. And AWS EC2 market place provides Centos 7.0 Image and when we create instance from this image, it applies security and patch updates resulting in Centos 7.3.

We can create our own Image of Centos 7.3 from existing servers but, It's always good be with the latest update for the OS for security reasons.

To finish it shortly, we have workarounds but not a solution yet 🙂 Thanks for your help.

I will update the solution which we follow.

Regards,

Vinay MP

[#6 · Accepted solution]

Hey All,

After series of tests, we decided to move to Centos 7.4 and Upgrade to HDP-2.6.3.0

With Centos 7.4 and Ambari Version 2.6.0.0, I don't see this issue eventhough I have 'Python 2.7.5'

With reference to my previous comment, it looks to be an Ambari issue.