Java/Python Updates and Ambari Agent TLS Settings

[#1]

JDK/Python Changes Causing Ambari Server/Agent Registration Issues

Recent JDK and Python updates have introduced behavior changes that can affect the Ambari Server to Ambari Agent registration process. The Ambari Server and Ambari Agent use TLS to register with each other securely. Recent changes to JDK and Python have been made to increase security by eliminating the use of insecure cipher suites and protocols. These changes ensure that more secure protocols and cipher suites are used by the Ambari Server when setting up its TLS sockets, and as a result, require the Ambari Agent’s Python client to be configured to use later versions of the TLS protocol to communicate with the Ambari Server.

You may be affected by these changes if you’ve recently upgraded your Python or JDK version and are noticing that your Ambari Agents are not heartbeating back to the Ambari Server and you see entries that resemble this in your Ambari Agent logs:

WARNING 2018-04-24 16:35:10,989 NetUtil.py:124 - Server at https://***.***.***.***:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-04-24 16:35:20,990 NetUtil.py:70 - Connecting to https://***.***.***.***:8440/ca
ERROR 2018-04-24 16:35:20,991 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-04-24 16:35:20,991 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.

When debugging this problem by using -Djavax.net.debug=all to start up the Ambari Server, you will also see the following in the Ambari Server logs:

qtp-ambari-agent-40, fatal error: 40: Client requested protocol TLSv1 not enabled or not supported
javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported
qtp-ambari-agent-40, SEND TLSv1.2 ALERT: fatal, description = handshake_failureqtp-ambari-agent-40, WRITE: TLSv1.2 Alert, length = 2
qtp-ambari-agent-40, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported

This is a telltale sign that the Ambari Agent is trying to communicate with the Ambari Server using TLSv1, instead of the TLS version mandated by upgraded JDK which is TLSv1.2.

There are two situations that have to be considered when solving this problem:

1.) If you are running CentOS 6 or SLES 11 the version of Python (2.6.x) does not work with TLSv1.2, so you must make changes to your newly updated JDK in order to proceed.

2.) If you are running CentOS 7, Debian 7, Ubuntu 14 & 16, or SLES 12 the version of Python (2.7.x) does work with TLS v1.2, so you only have to make changes to the Ambari Agent configuration to tell it to use TLS v1.2 in order to proceed.

Solution For CentOS 7, Debian 7, Ubuntu 14 & 16, or SLES 12 (Python 2.7)

To solve this problem simply configure the Ambari Agent to use TLSv1.2 when communicating with the Ambari Server by editing each Ambari Agent’s /etc/ambari-agent/conf/ambari-agent.ini file and adding the following configuration property to the security section:

[security]
force_https_protocol=PROTOCOL_TLSv1_2

Once this configuration change has been made, the Ambari Agent needs to be restarted. After restarting you should no longer see the ERROR’s in the Ambari Agent logs, and in the Ambari Server UI you’ll notice that all Ambari Agents are once again heartbeating.

Solution for CentOS 6, or SLES 11 (Python 2.6)

In this scenario, the preferred and most secure path forward is to upgrade your Operating System to a version that supports Python 2.7 if that cannot be done the only way forward is to weaken the security of your JDK installation by editting the java.security file in the JDK being used by the Ambari Server and make the following changes:

• Locate the jdk.tls.disabledAlgorithms property and remove the 3DES_EDE_CBC reference
• Save the file, and restart the Ambari Server

If there are any other questions, or special circumstances not covered in this post, please create a support ticket with Hortonworks Support.

[#2]

Is this fixed in Ambari 2.6.2 and Ambari 2.7 (for the CentOS7 and related versions)?

[#3]

Thanks Paul! Above solution for RHEL 7 resolved the issue after adding the entry in config file.

[#4]

Thanks Paul this article helped me resolve the issue

[#5]

Thanks for this solution. Worked for me when I used "Solution for CentOS 6". Even though I had python 2.7 installed by default on the machine I could not solve the issue with "Solution For CentOS 7". But changing security configuration "jdk.tls.disabledAlgorithms" for JDK as suggested resolved the issue.

[#6]

Hello, I am running into the same issue.

We use Centos 7.5 version on our HDP cluster nodes and RHEL updates ran on the ambari-server few days ago, and now I could not access our ambari-server's URL in the browser, as the connection gets dropped. And I see the following in the Ambari-server's logs:

javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

And when I run curl to the Ambari-Server's HTTPS url:

* Initializing NSS with certpath: sql:/etc/pki/nssdb
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0

My current environment: Ambari Server 2.4.0 for HDP 2.5.0.

Python version 2.7.5-69.el7_5 and Openssl version was updated in this server update, to 1.0.2k-12.el7

Should I downgrade to a different build of Python 2.7.5?

I always had Python Cert verification disabled, that did not help.

Other Options tried:

a) Updated the /etc/ambari-server/conf/ambari.properties with the following line:

security.server.disabled.protocols=SSL|SSLv2|SSLv2Hello|SSLv3|TLSv1

b) Updated the /etc/ambari-agent/conf/ambari-agent.ini with the following line:

force_https_protocol=PROTOCOL_TLSv1_2

Restarted the Ambari-server and the agent after the above update to configuration files, still not working.

Kindly please advise.

[#7]

This helped resolve our issue of spiking on a migration from oracle jdk 8 to open jdk 8 on centos 6.9

[#8]

Thanks it really fixes the problem RHEL 6.10 (Open JDK 1.8)