Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can we disable TLSv1 cipher for the Cloudera Platform?

avatar
New Contributor

We need to disable the TLSv1 cipher to meet a mandated organization security policy.

 

We've done this by editing the java.security file in the JDK 1.7 JRE's file and disabling other ciphers, RC4 and ECDHE (SSLv3 already disabled)

# Example:

jdk.tls.disabledAlgorithms=SSLv3, RC4, TLSv1, ECDHE

 

However when we disabled TLSv1 as well the whole stack wasn’t working and getting messages like this (I know it’s a warning)

2015-03-14 13:36:37,389 WARN 257524672@agentServer-0:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 
2015-03-14 13:36:37,395 WARN 732163666@scm-web-453:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 
2015-03-14 13:36:37,408 WARN 732163666@scm-web-453:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 

 

We've had to re-enable TLSv1 and every thing was working

# Example:

jdk.tls.disabledAlgorithms=SSLv3, RC4, ECDHE

 

Questions:

1. Can the cloudera platform function with TLSv1 disabled?

2. If it can, what is the recommended way to disable it?

 

19 REPLIES 19

avatar
Mentor
This would happen if any of your Java based daemons (including the CM server) are running on JDK6. Forcing the daemons to run on JDK7 will resolve the issue (you can remove JDK6 to enforce this, or use JAVA_HOME explicitly in /etc/default/cloudera-scm-server to point to JDK7).

avatar
New Contributor

Hi Harsh,

 

We are trying to disable TLS1.0 and 1.1. I can find a doc or blog which talks about the documented procedure to disable lower versions of TLS. Could you point me or share the documentation.

avatar
Master Guru

@AppaRao,

 

Thanks for bringing this up.  In the future, it would be a good idea to create a new thread rather than continue on an older one like this.  When that initial message was posted, I think Cloudera Manager's agent was not able to use anything but TLSv1 (which has since been fixed).

 

Cloudera is working on documenting this information.  For now, I'll add in what we have.

 

NOTE:

 

- You must have CDH 5.13.1 or higher in order for these steps to work for you

- We have only tested so far on Redhat and CentOS 6 and 7

 

----------------------------------------------------------------------------------------------------------------

 

Java Based Components

The vast majority of the components in the ecosystem are written in Java. This includes Accumulo, Flume, HBase, HDFS, Hive, Kafka, the Keystore Indexer, MapReduce, Oozie, Sentry, Solr, Spark, Sqoop, Yarn, and ZooKeeper. What you need to do is to disable the ciphers on Java which then applied to most of the Java-based services once they are restarted.

Fortunately, there is one universal place to enforce TLS usage for all Java-based components.  This is the "java.security" file.

Step 1: Update java.security

  • Open $JAVA_HOME/jre/lib/security/java.security in an editor
  • Add or replace this line: 
jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, MD5withRSA, DH keySize < 768, 3DES_EDE_CBC

NOTE:

  • This must be done on every machine in the cluster. Whenever Java is upgraded, this step must be performed again. 


HUE

The components of interest in Hue are the Hue load balancer and the Hue servers. We recommend the use of the Hue load balancer because more cipher suites will be available, and there is no issue with Firefox (as described below).

This first change is to the Hue load balancer.

Step 2: The Hue load balancer
On the CM machine, edit /usr/share/cmf/hue/httpd/httpd.conf

Edit the SSLProtocol line to look like this:

 

SSLProtocol             -all +TLSv1.2 

NOTE: This step also has to be performed after each CM upgrade.  

Step 3: The Hue Server

In CM, add the following to the “Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini”

 

[desktop]
ssl_cipher_list=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!SSLv3:!TLSv1


NOTE: The Firefox web browser does not support any cipher suites that overlap with these ciphers. As a result, the Firefox web browser will not load the page and will display the error “SSL_ERROR_NO_CYPHER_OVERLAP”.  Chrome, IE, Edge, and Safari do not have this issue, and are fine. This issue does not exist for the Hue Load Balancer above.

Impala

There are two different mechanisms to get TLS 1.2 support, depending on your operating system.  On RHEL/CentOS 7, add the following to a CM Configuration Snippet (Safety Valve).

Step 4a: Impala on RHEL/CentOS 7
In CM, add the following parameter in Impala's safety valve: Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)

 

-ssl_minimum_version=tlsv1.2

On RHEL/CentOS 6, the above flag unfortunately does not work. Add the following instead:

Step 4b: Impala on RHEL/CentOS 6
In CM, add the following parameter in Impala's safety valve: Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)

 

-ssl_cipher_list=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!SSLv3:!TLS1

NOTE: This solution has the same caveats with respect to Firefox as described above for Hue.

Kudu

There are two ports to configure for Kudu, the rpc protocol port and the webserver protocol port. In CDH 5.13.1 it’s possible to restrict the TLS protocol to TLS 1.2 for the rpc protocol port. This is the port where that all the data travels through. Unfortunately it’s not possible to similarly enforce TLS 1.2 on the web server port in CDH 5.13.1. The traffic that goes over the web server port is of a generally non-sensitive nature - like status.

Like Impala, there are two different solutions depending on the OS version. For RHEL/CentOS 7, do this:

Step 5a: Kudu on RHEL/CentOS 7
In CM, add the following parameter to the “Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile”

 

-rpc_tls_min_protocol=TLSv1.2

On RHEL/CentOS 6, add the following instead:

Step 5b: Kudu on RHEL/CentOS 6
In CM, add the following parameter to the “Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile”

 

-rpc_tls_ciphers=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!SSLv3:!TLSv1

NOTE: Once again, this solution for RHEL/CentOS 6 has the same caveats with respect to Firefox as described above for Hue.

Restart

Now you’ll want to restart everything.

Step 6: Restart

  • In CM, restart all the affected clusters
  • In CM, restart the Cloudera Management Services
  • On the CM server machine, restart CM itself using
sudo service cloudera-scm-server restart
 
 
 

avatar
Contributor

Is there any option for the ssl_cipher_list for Cloudera Manager Agents (Port 9000) to support stronger encryptions?  We are on CDH 6.2.

avatar
Master Guru

@BrettM,

 

We have a fix to disable TLSv1 that is already slated for upcoming releases of 6.2.x and 6.3.

I tested, though, and found that the fix still allowed TLSv1_1.  After some research I was able to work around a limitation with M2Crypto (which does not support the SSL_OP_NO_TLSv1_1 flag that OpenSSL uses to disable TLSv1_1).

I opened a new Jira internally at Cloudera so we can review it and check it in as needed.

 

If you are comfortable editing python files, you can test it out yourself.

 

  1. Back up /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py

    NOTE:  if you are on an el6 platform, the above path may differ.  To find it, you can run:
    # find /opt/cloudera/cm-agent -name "https.py"
  2. Edit the https.py file (on el7 platforms, it will be /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py
  3. Find the line that looks like this:

    ctx.set_options(m2.SSL_OP_NO_SSLv2 | m2.SSL_OP_NO_SSLv3)

  4. Edit the line to look like this (additions in bold):

    ctx.set_options(m2.SSL_OP_NO_TLSv1 | 0x10000000L | m2.SSL_OP_NO_SSLv2 | m2.SSL_OP_NO_SSLv3)

    NOTE:  This is Python, so make sure the indentation does not change or the file may not compile during runtime
  5. Restart the agent with

    # service cloudera-scm-agent restart

  6. Test by running this on the same host where you made the above edit:

    #openssl s_client -connect `hostname -f`:9000 -tls1_1 -msg < /dev/null

    The expected result is that after the ClientHello, nothing is returned from port 9000.  It might look like this:

    CONNECTED(00000003)
    >>> TLS 1.1 Handshake [length 0096], ClientHello
    01 00 00 92 03 02 5d 29 1e 3b c2 2a 7c 95 49 36
    87 e5 8e 82 31 cb af 9a 64 6d a2 1d 2a 2a 95 9f
    76 13 3a aa d3 24 00 00 4c c0 14 c0 0a 00 39 00
    38 00 88 00 87 c0 0f c0 05 00 35 00 84 c0 13 c0
    09 00 33 00 32 00 9a 00 99 00 45 00 44 c0 0e c0
    04 00 2f 00 96 00 41 c0 12 c0 08 00 16 00 13 c0
    0d c0 03 00 0a 00 07 c0 11 c0 07 c0 0c c0 02 00
    05 00 04 00 ff 01 00 00 1d 00 0b 00 04 03 00 01
    02 00 0a 00 08 00 06 00 19 00 18 00 17 00 23 00
    00 00 0f 00 01 01

If that is what you see, TLSv1 and TLSv1.1 connections will not be serviced.  The "hang" is OK in this case since Cloudera Manager is the only client of port 9000 and it should be using TLSv1.2.

 

WARNING:  The above steps I tested on my own host, but use at your own risk.  If you prefer to not incur the risk, please await an upcoming release that includes the code changes.  If for some reason the agent doesn't restart, you can copy the backup file in place and restart the agent to recover.  This will not impact data.

 

Cheers,

 

Ben

avatar
Master Guru

Oops; I forgot ot mention that the line to edit should be 95 or close to there.

Also, this would need to be done on all hosts managed by Cloudera Manager as each agent listens on port 9000.

 

 

avatar
Contributor

@bgooley 

 

This information is incredibly helpful and insightful.  We will try this out solution.  Thank you very much!

 

Thanks,

Brett

avatar
Contributor

@bgooley 

 

That worked great.  I ran this command and the TLS1_1 just hung as you suggested.  Thank You!

 

> openssl s_client -connect <server_name>:9000 -tls1_1 -msg < /dev/null
CONNECTED(00000003)
>>> TLS 1.1 Handshake [length 00a2], ClientHello
01 00 00 9e 03 02 5d 32 57 4b 04 0a af 47 03 90
....

avatar
Master Guru

@BrettM,

 

That's great to hear.  I'll inform the Cloudera Manager engineering team.