How can we disable TLSv1 cipher for the Cloudera Platform?

[#1]

We need to disable the TLSv1 cipher to meet a mandated organization security policy.

We've done this by editing the java.security file in the JDK 1.7 JRE's file and disabling other ciphers, RC4 and ECDHE (SSLv3 already disabled)

# Example:

jdk.tls.disabledAlgorithms=SSLv3, RC4, TLSv1, ECDHE

However when we disabled TLSv1 as well the whole stack wasn’t working and getting messages like this (I know it’s a warning)

2015-03-14 13:36:37,389 WARN 257524672@agentServer-0:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 
2015-03-14 13:36:37,395 WARN 732163666@scm-web-453:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 
2015-03-14 13:36:37,408 WARN 732163666@scm-web-453:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 

We've had to re-enable TLSv1 and every thing was working

# Example:

jdk.tls.disabledAlgorithms=SSLv3, RC4, ECDHE

Questions:

1. Can the cloudera platform function with TLSv1 disabled?

2. If it can, what is the recommended way to disable it?

[#2]

This would happen if any of your Java based daemons (including the CM server) are running on JDK6. Forcing the daemons to run on JDK7 will resolve the issue (you can remove JDK6 to enforce this, or use JAVA_HOME explicitly in /etc/default/cloudera-scm-server to point to JDK7).

[#3]

Hi Harsh,

We are trying to disable TLS1.0 and 1.1. I can find a doc or blog which talks about the documented procedure to disable lower versions of TLS. Could you point me or share the documentation.

[#4]

@AppaRao,

Thanks for bringing this up.  In the future, it would be a good idea to create a new thread rather than continue on an older one like this.  When that initial message was posted, I think Cloudera Manager's agent was not able to use anything but TLSv1 (which has since been fixed).

Cloudera is working on documenting this information.  For now, I'll add in what we have.

NOTE:

- You must have CDH 5.13.1 or higher in order for these steps to work for you

- We have only tested so far on Redhat and CentOS 6 and 7

----------------------------------------------------------------------------------------------------------------

Java Based Components

The vast majority of the components in the ecosystem are written in Java. This includes Accumulo, Flume, HBase, HDFS, Hive, Kafka, the Keystore Indexer, MapReduce, Oozie, Sentry, Solr, Spark, Sqoop, Yarn, and ZooKeeper. What you need to do is to disable the ciphers on Java which then applied to most of the Java-based services once they are restarted.

Fortunately, there is one universal place to enforce TLS usage for all Java-based components.  This is the "java.security" file.

Step 1: Update java.security

• Open $JAVA_HOME/jre/lib/security/java.security in an editor
• Add or replace this line: 

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, MD5withRSA, DH keySize < 768, 3DES_EDE_CBC

NOTE:

• This must be done on every machine in the cluster. Whenever Java is upgraded, this step must be performed again. 

HUE

The components of interest in Hue are the Hue load balancer and the Hue servers. We recommend the use of the Hue load balancer because more cipher suites will be available, and there is no issue with Firefox (as described below).

This first change is to the Hue load balancer.

Step 2: The Hue load balancer
On the CM machine, edit /usr/share/cmf/hue/httpd/httpd.conf

Edit the SSLProtocol line to look like this:

SSLProtocol             -all +TLSv1.2 

NOTE: This step also has to be performed after each CM upgrade.  

Step 3: The Hue Server

In CM, add the following to the “Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini”

[desktop]
ssl_cipher_list=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!SSLv3:!TLSv1

NOTE: The Firefox web browser does not support any cipher suites that overlap with these ciphers. As a result, the Firefox web browser will not load the page and will display the error “SSL_ERROR_NO_CYPHER_OVERLAP”.  Chrome, IE, Edge, and Safari do not have this issue, and are fine. This issue does not exist for the Hue Load Balancer above.

Impala

There are two different mechanisms to get TLS 1.2 support, depending on your operating system.  On RHEL/CentOS 7, add the following to a CM Configuration Snippet (Safety Valve).

Step 4a: Impala on RHEL/CentOS 7
In CM, add the following parameter in Impala's safety valve: Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)

-ssl_minimum_version=tlsv1.2

On RHEL/CentOS 6, the above flag unfortunately does not work. Add the following instead:

Step 4b: Impala on RHEL/CentOS 6
In CM, add the following parameter in Impala's safety valve: Impala Command Line Argument Advanced Configuration Snippet (Safety Valve)

-ssl_cipher_list=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!SSLv3:!TLS1

NOTE: This solution has the same caveats with respect to Firefox as described above for Hue.

Kudu

There are two ports to configure for Kudu, the rpc protocol port and the webserver protocol port. In CDH 5.13.1 it’s possible to restrict the TLS protocol to TLS 1.2 for the rpc protocol port. This is the port where that all the data travels through. Unfortunately it’s not possible to similarly enforce TLS 1.2 on the web server port in CDH 5.13.1. The traffic that goes over the web server port is of a generally non-sensitive nature - like status.

Like Impala, there are two different solutions depending on the OS version. For RHEL/CentOS 7, do this:

Step 5a: Kudu on RHEL/CentOS 7
In CM, add the following parameter to the “Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile”

-rpc_tls_min_protocol=TLSv1.2

On RHEL/CentOS 6, add the following instead:

Step 5b: Kudu on RHEL/CentOS 6
In CM, add the following parameter to the “Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile”

-rpc_tls_ciphers=DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!SSLv3:!TLSv1

NOTE: Once again, this solution for RHEL/CentOS 6 has the same caveats with respect to Firefox as described above for Hue.

Restart

Now you’ll want to restart everything.

Step 6: Restart

• In CM, restart all the affected clusters
• In CM, restart the Cloudera Management Services
• On the CM server machine, restart CM itself using

sudo service cloudera-scm-server restart

[#5]

Is there any option for the ssl_cipher_list for Cloudera Manager Agents (Port 9000) to support stronger encryptions?  We are on CDH 6.2.

[#6]

@BrettM,

We have a fix to disable TLSv1 that is already slated for upcoming releases of 6.2.x and 6.3.

I tested, though, and found that the fix still allowed TLSv1_1.  After some research I was able to work around a limitation with M2Crypto (which does not support the SSL_OP_NO_TLSv1_1 flag that OpenSSL uses to disable TLSv1_1).

I opened a new Jira internally at Cloudera so we can review it and check it in as needed.

If you are comfortable editing python files, you can test it out yourself.

1. Back up /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py
   
   NOTE:  if you are on an el6 platform, the above path may differ.  To find it, you can run:
   # find /opt/cloudera/cm-agent -name "https.py"
2. Edit the https.py file (on el7 platforms, it will be /opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py
3. Find the line that looks like this:
   
   ctx.set_options(m2.SSL_OP_NO_SSLv2 | m2.SSL_OP_NO_SSLv3)
   
   
4. Edit the line to look like this (additions in bold):
   
   ctx.set_options(m2.SSL_OP_NO_TLSv1 | 0x10000000L | m2.SSL_OP_NO_SSLv2 | m2.SSL_OP_NO_SSLv3)
   
   NOTE:  This is Python, so make sure the indentation does not change or the file may not compile during runtime
5. Restart the agent with
   
   # service cloudera-scm-agent restart
   
   
6. Test by running this on the same host where you made the above edit:
   
   #openssl s_client -connect `hostname -f`:9000 -tls1_1 -msg < /dev/null
   
   The expected result is that after the ClientHello, nothing is returned from port 9000.  It might look like this:
   
   CONNECTED(00000003)
   >>> TLS 1.1 Handshake [length 0096], ClientHello
   01 00 00 92 03 02 5d 29 1e 3b c2 2a 7c 95 49 36
   87 e5 8e 82 31 cb af 9a 64 6d a2 1d 2a 2a 95 9f
   76 13 3a aa d3 24 00 00 4c c0 14 c0 0a 00 39 00
   38 00 88 00 87 c0 0f c0 05 00 35 00 84 c0 13 c0
   09 00 33 00 32 00 9a 00 99 00 45 00 44 c0 0e c0
   04 00 2f 00 96 00 41 c0 12 c0 08 00 16 00 13 c0
   0d c0 03 00 0a 00 07 c0 11 c0 07 c0 0c c0 02 00
   05 00 04 00 ff 01 00 00 1d 00 0b 00 04 03 00 01
   02 00 0a 00 08 00 06 00 19 00 18 00 17 00 23 00
   00 00 0f 00 01 01

If that is what you see, TLSv1 and TLSv1.1 connections will not be serviced.  The "hang" is OK in this case since Cloudera Manager is the only client of port 9000 and it should be using TLSv1.2.

WARNING:  The above steps I tested on my own host, but use at your own risk.  If you prefer to not incur the risk, please await an upcoming release that includes the code changes.  If for some reason the agent doesn't restart, you can copy the backup file in place and restart the agent to recover.  This will not impact data.

Cheers,

Ben

[#7]

Oops; I forgot ot mention that the line to edit should be 95 or close to there.

Also, this would need to be done on all hosts managed by Cloudera Manager as each agent listens on port 9000.

[#8]

@bgooley 

This information is incredibly helpful and insightful.  We will try this out solution.  Thank you very much!

Thanks,

Brett

[#9]

@bgooley 

That worked great.  I ran this command and the TLS1_1 just hung as you suggested.  Thank You!

> openssl s_client -connect <server_name>:9000 -tls1_1 -msg < /dev/null
CONNECTED(00000003)
>>> TLS 1.1 Handshake [length 00a2], ClientHello
01 00 00 9e 03 02 5d 32 57 4b 04 0a af 47 03 90
....

[#10]

@BrettM,

That's great to hear.  I'll inform the Cloudera Manager engineering team.

[#11]

@bgooley 

So we had another scan on the system and the following two vulnerabilities are still showing up on Port 9000 even after we put in that TLS v1.1 fix. 

1. https://www.tenable.com/plugins/nessus/42873  and described here: https://www.openssl.org/blog/blog/2016/08/24/sweet32/

2.   https://www.tenable.com/plugins/nnm/7282  also described here: https://www.secpod.com/blog/cve-2015-2808-bar-mitzvah-attack-in-rc4-2/

For item #1, none of these commands returned a success:

openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -ssl2
openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -ssl3
openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -tls1
openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -tls1_1

For item #1, this command returned a success:
openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -tls1_2

For item #2, I can run this command it shows RC4 cipher is being supported on port 9000

openssl s_client -cipher RC4 -connect <servername>:9000 -msg < /dev/null

Please advise.

Thanks,

Brett

[#12]

Hi @BrettM ,

Thank you once again for pointing out a security issue!

I ran nmap with ssl-enum-ciphers and saw the same problems.  I took a stab and found that M2Crypto does let you set ciphers.  I chose a set that limits to good ciphers.

As before, you can test the fix I proposed by editing the "https.py" file on each host.

On an el7 host, the path will likely be:

/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py

(1)

Back up the https.py file so you can revert if there is a problem starting the agent

(2)

Below the line you edited:

ctx.set_options(m2.SSL_OP_NO_TLSv1 | 0x10000000L | m2.SSL_OP_NO_SSLv2 | m2.SSL_OP_NO_SSLv3)

add this:

ctx.set_cipher_list("HIGH:!DSS:!DH:!ADH:!DES:!3DES:!SHA1:!aNULL:!eNULL:!EXPORT:!SSLv2:!SSLv3:!TLSv1")

(3)

Save your change

Restart the agent with:

service cloudera-scm-agent restart

(4)

Test to see if any vulnerable ciphers are still allowed.
As before, if the change helps in your deployment, this would need to be applied on all hosts where the agent runs.  Any upgrades of CM that do not include fixes would also require you to edit manually.

I have already created another Jira for this with Cloudera Manager engineering.

[#13]

Hi @bgooley,

We applied that additional line of code to the "https.py" file and our scan came back clean on port 9000 for these two vulnerabilities:

1. https://www.tenable.com/plugins/nessus/42873  and described here: https://www.openssl.org/blog/blog/2016/08/24/sweet32/

2.   https://www.tenable.com/plugins/nnm/7282  also described here: https://www.secpod.com/blog/cve-2015-2808-bar-mitzvah-attack-in-rc4-2/

Thank you very much!

Do you have any insight into when both of these updates to the "https.py" will be GA?

Thanks,

Brett

[#14]

@BrettM,

The openssl cipher list I provided was an example that worked for me, basically.  What ciphers did your scan report to be vulnerable?  The cipher list can be modified to suit your needs if necessary.

I am surprised, though, since I thought the openssl string I provided was able to resolve for me.

Knowing exactly which ciphers were flagged by nessus would help us understand perhaps.

As for a fix in the product, we need to evaluate the changes, get the approved for release, then run then through testing, so it will be a while yet.

[#15]

@bgooley ,

To be honest, I only got this information below and don't 100% understand it because I am not security expert. This is might make more sense to you:

Vulnerability #1: SSL Medium Strength Cipher Suites Supported (SWEET32)

Medium Strength Ciphers (> 64-bit and < 112-bit key or 3DES)

KRB5-DES-CBC3-MD5 Kx=KRB5 Au=KRB5 Enc=3DES-CBC(168) Mac=MD5 

KRB5-DES-CBC3-SHA Kx=KRB5 Au=KRB5 Enc=3DES-CBC(168) Mac=SHA1 

DES-CBC3-SHA Kx=RSA Au=RSA Enc=3DES-CBC(168) Mac=SHA1 

The fields above are :

{OpenSSL ciphername}

Kx={key exchange}

Au={authentication}

Enc={symmetric encryption method}

Mac={message authentication code}

Vulnerability #2: SSL RC4 Cipher Suites Supported (Bar Mitzvah)

List of RC4 cipher suites supported by the remote server :

High Strength Ciphers (>= 112-bit key)

KRB5-RC4-MD5 Kx=KRB5 Au=KRB5 Enc=RC4(128) Mac=MD5

KRB5-RC4-SHA Kx=KRB5 Au=KRB5 Enc=RC4(128) Mac=SHA1

RC4-MD5 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5

RC4-SHA Kx=RSA Au=RSA Enc=RC4(128) Mac=SHA1

The fields above are :

{OpenSSL ciphername}

Kx={key exchange}

Au={authentication}

Enc={symmetric encryption method}

Mac={message authentication code}

I hope this means something to you. 🙂

Thanks,

Brett

[#16]

@BrettM,

This makes me think that the code change did not actually take effect since the cipher list I gave you as an example removes DES and 3DES ciphers.  This worked in my environment so the ciphers that were available before the change were no longer available afterward.

Can you confirm that the agent Nessus is detecting as having vulnerable ciphers has the following code changes in https.py:

  ctx = SSL.Context('sslv23')
  ctx.set_options(m2.SSL_OP_NO_TLSv1 | 0x10000000L | m2.SSL_OP_NO_SSLv2 | m2.SSL_OP_NO_SSLv3)
  ctx.set_cipher_list("HIGH:!DSS:!DH:!ADH:!DES:!3DES:!SHA1:!RC4:!aNULL:!eNULL:!EXPORT:!SSLv2:!SSLv3:!TLSv1")
  ctx.set_session_id_ctx("cm-agent")

I added "!RC4" in there just in case but really "HIGH" should have taken care of that I think.

Also, be sure you restarted the agent after the last change to make sure the changes took place.

[#17]

@bgooley 

Our scan back from Nessus is good on Port 9000. We are not getting any of those aforementioned issues with the previous update from last week you provided. 

I was just letting you know the original message I was getting back about Port 9000. I agree the !HIGH would have removed the RC4 cipher.

We are good, this issue on Port 9000 is remediated.  Thank You very much!

Thanks,

Brett

[#18]

@bgooley 

In CDH 6.3.x, this appears to have changed and the "https.py" file is slightly different now.  It accepts the cipher_list as a configuration item.  The way we secured Port 900 is by doing these

steps:

1) Check to see if RC4 (and other weak ciphers) are open on Port 9000:

openssl s_client -cipher RC4 -connect <server>:9000 -msg

2) Edit the "/etc/cloudera-scm-agent/config.ini" file

3) Under the "[Security]" section of the config.ini file, we added these lines:

# Custom Cipher List to close vulnerabilities for port 9000
cipher_list=HIGH:!DSS:!DH:!ADH:!DES:!3DES:!SHA1:!RC4:!aNULL:!eNULL:!EXPORT:!SSLv2:!SSLv3:!TLSv1

4) Restart the Cloudera CM-Agent:  

sudo service cloudera-scm-agent restart

5) Wait a minute or so and then rerun the OpenSSL command and RC4 (and other weak ciphers, if you test them) are closed:

openssl s_client -cipher RC4 -connect <server>:9000 -msg

It would be great if Cloudera could add this to their documentation on how to add this additional security to the CM Agent.

[#19]

@bgooley 

That was quite a good list of steps I could find after searching a lot for procedures on upgrading the TLS 1.1 to 1.2.

I actually applied these steps on one of our test environment  on CDH 5.13 cluster on centos 6 wihin our organization and submitted for the vulnerability scan and the report has come up with quite a number of ports still have TLS 1.1 and are vulnerable

These are the ports:

11371 -- KTS server

11381-- postgresssql database

50475  External -  Datanode-- dfs.datanode.https.address

13562  Yarn //mapreduce.shuffle.port

9093--kafka

8985 -- solr_https_port

8044 --Yarn,node manager --yarn. nodemanager. webapp.https.address

20550 --hbase.rest.port

19890-- Yarn Job history server, mapreduce. jobhistory. webapp.address

11443 -- Oozie server

9095 --Hbase Thrift server

8889 -- Hue load balancer

60010 -- hbase.master. info.port (http)

7187-- Cloudera manager server (metadataserve/https web UI)

50470 -- dfs.https.address or dfs.namenode.https-address (dfs.https.addressis deprecated (but still works)

14000 -- HttpFS

8481 --Hadoop --dfs.journalnode. https-address

8090 -- yarn. resourcemanager. webapp.https.address

8044 ---yarn. nodemanager. webapp.https.address

60030 -- hbase. regionserver. info.port.

please let us know how we can overcome/resolve this issue.

Thanks

Suresh

.

[#20]

@bgooley 

Hi

That was quite a good list of steps I could find after searching a lot for procedures on upgrading the TLS 1.1 to 1.2.

I actually applied these steps on one of our test environment on CDH 5.13 cluster on centos 6 within our organization and submitted for the vulnerability scan and the report has come up with quite a number of ports still have TLS 1.1 vulnerability.

These are the ports:

11371 -- KTS server

11381-- postgresssql database

50475 External - Datanode-- dfs.datanode.https.address

13562 Yarn //mapreduce.shuffle.port

9093--kafka

8985 -- solr_https_port

8044 --Yarn,node manager --yarn. nodemanager. webapp.https.address

20550 --hbase.rest.port

19890-- Yarn Job history server, mapreduce. jobhistory. webapp.address

11443 -- Oozie server

9095 --Hbase Thrift server

8889 -- Hue load balancer

60010 -- hbase.master. info.port (http)

7187-- Cloudera manager server (metadataserve/https web UI)

50470 -- dfs.https.address or dfs.namenode.https-address (dfs.https.addressis deprecated (but still works)

14000 -- HttpFS

8481 --Hadoop --dfs.journalnode. https-address

8090 -- yarn. resourcemanager. webapp.https.address

8044 ---yarn. nodemanager. webapp.https.address

60030 -- hbase. regionserver. info.port.

please let us know how we can overcome/resolve this issue. Looking forward for your response.

Thanks

Abdul