Python 2.7.5-69 compatability with HDP 2.6.4

[#1]

We are running RHEL 7, HDP 2.6.4, and Ambari 2.6.2.0 on a 6 VM cluster. This morning we patched the cluster and one of the items it upgraded was Python. It took Python from 2.7.5-68 to 2.7.5-69, effectively breaking communication between the Ambari Server and Agents. The following error was thrown every 10 seconds in the ambari-agent.log

INFO 2018-09-06 13:09:36,133 NetUtil.py:70 - Connecting to https://{HOST_NAME}:8440/ca
ERROR 2018-09-06 13:09:36,166 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-09-06 13:09:36,167 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

My question is, what version of HDP/Ambari do I need to be on in order to use the latest version of Python?

Thanks,

Andrew

[#2 · Accepted solution]

@Andrew Mills

Regarding the following error

ERROR 2018-09-06 13:09:36,166 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

You can get more detailed explanation on why do we see this error and how to remediate it here: https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.h...

Summary:

This can happen when Ambari Agent is trying to communicate with the Ambari Server using TLSv1, instead of the TLS version mandated by upgraded JDK which is TLSv1.2.

There are two situations that have to be considered when solving this problem:

1.) If you are running CentOS 6 or SLES 11 the version of Python (2.6.x) does not work with TLSv1.2, so you must make changes to your newly updated JDK in order to proceed.

2.) If you are running CentOS 7, Debian 7, Ubuntu 14 & 16, or SLES 12 the version of Python (2.7.x) does work with TLS v1.2, so you only have to make changes to the Ambari Agent configuration to tell it to use TLS v1.2 in order to proceed.

Solution For CentOS 7, Debian 7, Ubuntu 14 & 16, or SLES 12 (Python 2.7)

To solve this problem simply configure the Ambari Agent to use TLSv1.2 when communicating with the Ambari Server by editing each Ambari Agent’s /etc/ambari-agent/conf/ambari-agent.ini file and adding the following configuration property to the security section:

[security]
force_https_protocol=PROTOCOL_TLSv1_2

Once this configuration change has been made, the Ambari Agent needs to be restarted. After restarting you should no longer see the ERROR’s in the Ambari Agent logs, and in the Ambari Server UI you’ll notice that all Ambari Agents are once again heartbeating.

Solution for CentOS 6, or SLES 11 (Python 2.6)

In this scenario the only way forward is to edit the java.security file in the JDK being used by the Ambari Server and make the following changes:

• Locate the jdk.tls.disabledAlgorithms property and remove the 3DES_EDE_CBC reference
• Save the file, and restart the Ambari Server

.

[#3]

@Andrew Mills

Regarding your query: "What version of HDP/Ambari do I need to be on in order to use the latest version of Python?"

As per the Support matrix you should be able to use :

For Ambari 2.6.2.x Any of the following Python version are supported as per: https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.0/bk_ambari-installation/content/mmsr_softwar...

Python
--------
For SLES 12 -->  Python 2.7.x
For CentOS 7, Ubuntu 14, Ubuntu 16, and Debian 9 --> Python 2.7.x

.

[#4]

@Jay Kumar SenSharma thank you for the answer, this fixed my issue. Adding force_https_protocol=PROTOCOL_TLSv1 to the Ambari Agents allowed them to connect to the server.

[#5]

@Jay Kumar SenSharma I am having a similar issue on Centos 7.x. Please see https://community.hortonworks.com/comments/222163/view.html. (scroll all the way down to the last comment for exact details of the issue)

Ambari-server is up and running, but Ambari Server UI is not accessible.

My curl request to the https://<ambari-server host>:8xxx fails with the below message:

* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file

Though all the ambari-agents in the cluster are updated to force the use of TLSV1_2, no luck.

Any thoughts? Thanks in advance!

[#6]

We were able to solve this by locating one of the Cipher Suite which is originally marked as 'disabled' in ambari.properties, and enabled the same by removing the ambari.properties and restarting the server.

First tried with removing the ciphers.disabled properties(take a backup), and then restart ambari-server. Used Openssl command to connect to the ambari-server on https port. Identified which cipher suite is being used to establish connection, and then located the corresponding RFC cipher mapping for the cipher suite and removed that in the list of cipher suites listed on the ciphers.disabled property in ambari.properties file.