Java/Python Updates and Ambari Agent TLS Settings

JDK/Python Changes Causing Ambari Server/Agent Registration Issues

Recent JDK and Python updates have introduced behavior changes that can affect the Ambari Server to Ambari Agent registration process. The Ambari Server and Ambari Agent use TLS to register with each other securely. Recent changes to JDK and Python have been made to increase security by eliminating the use of insecure cipher suites and protocols. These changes ensure that more secure protocols and cipher suites are used by the Ambari Server when setting up its TLS sockets, and as a result, require the Ambari Agent’s Python client to be configured to use later versions of the TLS protocol to communicate with the Ambari Server.

You may be affected by these changes if you’ve recently upgraded your Python or JDK version and are noticing that your Ambari Agents are not heartbeating back to the Ambari Server and you see entries that resemble this in your Ambari Agent logs:

WARNING 2018-04-24 16:35:10,989 NetUtil.py:124 - Server at https://***.***.***.***:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-04-24 16:35:20,990 NetUtil.py:70 - Connecting to https://***.***.***.***:8440/ca
ERROR 2018-04-24 16:35:20,991 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-04-24 16:35:20,991 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.

When debugging this problem by using -Djavax.net.debug=all to start up the Ambari Server, you will also see the following in the Ambari Server logs:

qtp-ambari-agent-40, fatal error: 40: Client requested protocol TLSv1 not enabled or not supported
javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported
qtp-ambari-agent-40, SEND TLSv1.2 ALERT: fatal, description = handshake_failureqtp-ambari-agent-40, WRITE: TLSv1.2 Alert, length = 2
qtp-ambari-agent-40, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported

This is a telltale sign that the Ambari Agent is trying to communicate with the Ambari Server using TLSv1, instead of the TLS version mandated by upgraded JDK which is TLSv1.2.

There are two situations that have to be considered when solving this problem:

1.) If you are running CentOS 6 or SLES 11 the version of Python (2.6.x) does not work with TLSv1.2, so you must make changes to your newly updated JDK in order to proceed.

2.) If you are running CentOS 7, Debian 7, Ubuntu 14 & 16, or SLES 12 the version of Python (2.7.x) does work with TLS v1.2, so you only have to make changes to the Ambari Agent configuration to tell it to use TLS v1.2 in order to proceed.

Solution For CentOS 7, Debian 7, Ubuntu 14 & 16, or SLES 12 (Python 2.7)

To solve this problem simply configure the Ambari Agent to use TLSv1.2 when communicating with the Ambari Server by editing each Ambari Agent’s /etc/ambari-agent/conf/ambari-agent.ini file and adding the following configuration property to the security section:

[security]
force_https_protocol=PROTOCOL_TLSv1_2

Once this configuration change has been made, the Ambari Agent needs to be restarted. After restarting you should no longer see the ERROR’s in the Ambari Agent logs, and in the Ambari Server UI you’ll notice that all Ambari Agents are once again heartbeating.

Solution for CentOS 6, or SLES 11 (Python 2.6)

In this scenario, the preferred and most secure path forward is to upgrade your Operating System to a version that supports Python 2.7 if that cannot be done the only way forward is to weaken the security of your JDK installation by editting the java.security file in the JDK being used by the Ambari Server and make the following changes:

• Locate the jdk.tls.disabledAlgorithms property and remove the 3DES_EDE_CBC reference
• Save the file, and restart the Ambari Server

If there are any other questions, or special circumstances not covered in this post, please create a support ticket with Hortonworks Support.