How can we disable TLSv1 cipher for the Cloudera Platform?

[#1]

We need to disable the TLSv1 cipher to meet a mandated organization security policy.

We've done this by editing the java.security file in the JDK 1.7 JRE's file and disabling other ciphers, RC4 and ECDHE (SSLv3 already disabled)

# Example:

jdk.tls.disabledAlgorithms=SSLv3, RC4, TLSv1, ECDHE

However when we disabled TLSv1 as well the whole stack wasn’t working and getting messages like this (I know it’s a warning)

2015-03-14 13:36:37,389 WARN 257524672@agentServer-0:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 
2015-03-14 13:36:37,395 WARN 732163666@scm-web-453:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 
2015-03-14 13:36:37,408 WARN 732163666@scm-web-453:org.mortbay.log: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported 

We've had to re-enable TLSv1 and every thing was working

# Example:

jdk.tls.disabledAlgorithms=SSLv3, RC4, ECDHE

Questions:

1. Can the cloudera platform function with TLSv1 disabled?

2. If it can, what is the recommended way to disable it?

[#2]

@bgooley 

So we had another scan on the system and the following two vulnerabilities are still showing up on Port 9000 even after we put in that TLS v1.1 fix. 

1. https://www.tenable.com/plugins/nessus/42873  and described here: https://www.openssl.org/blog/blog/2016/08/24/sweet32/

2.   https://www.tenable.com/plugins/nnm/7282  also described here: https://www.secpod.com/blog/cve-2015-2808-bar-mitzvah-attack-in-rc4-2/

For item #1, none of these commands returned a success:

openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -ssl2
openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -ssl3
openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -tls1
openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -tls1_1

For item #1, this command returned a success:
openssl s_client -connect <servername>:9000 -cipher "DES:3DES" -tls1_2

For item #2, I can run this command it shows RC4 cipher is being supported on port 9000

openssl s_client -cipher RC4 -connect <servername>:9000 -msg < /dev/null

Please advise.

Thanks,

Brett

[#3]

Hi @BrettM ,

Thank you once again for pointing out a security issue!

I ran nmap with ssl-enum-ciphers and saw the same problems.  I took a stab and found that M2Crypto does let you set ciphers.  I chose a set that limits to good ciphers.

As before, you can test the fix I proposed by editing the "https.py" file on each host.

On an el7 host, the path will likely be:

/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py

(1)

Back up the https.py file so you can revert if there is a problem starting the agent

(2)

Below the line you edited:

ctx.set_options(m2.SSL_OP_NO_TLSv1 | 0x10000000L | m2.SSL_OP_NO_SSLv2 | m2.SSL_OP_NO_SSLv3)

add this:

ctx.set_cipher_list("HIGH:!DSS:!DH:!ADH:!DES:!3DES:!SHA1:!aNULL:!eNULL:!EXPORT:!SSLv2:!SSLv3:!TLSv1")

(3)

Save your change

Restart the agent with:

service cloudera-scm-agent restart

(4)

Test to see if any vulnerable ciphers are still allowed.
As before, if the change helps in your deployment, this would need to be applied on all hosts where the agent runs.  Any upgrades of CM that do not include fixes would also require you to edit manually.

I have already created another Jira for this with Cloudera Manager engineering.

[#4]

Hi @bgooley,

We applied that additional line of code to the "https.py" file and our scan came back clean on port 9000 for these two vulnerabilities:

1. https://www.tenable.com/plugins/nessus/42873  and described here: https://www.openssl.org/blog/blog/2016/08/24/sweet32/

2.   https://www.tenable.com/plugins/nnm/7282  also described here: https://www.secpod.com/blog/cve-2015-2808-bar-mitzvah-attack-in-rc4-2/

Thank you very much!

Do you have any insight into when both of these updates to the "https.py" will be GA?

Thanks,

Brett

[#5]

@BrettM,

The openssl cipher list I provided was an example that worked for me, basically.  What ciphers did your scan report to be vulnerable?  The cipher list can be modified to suit your needs if necessary.

I am surprised, though, since I thought the openssl string I provided was able to resolve for me.

Knowing exactly which ciphers were flagged by nessus would help us understand perhaps.

As for a fix in the product, we need to evaluate the changes, get the approved for release, then run then through testing, so it will be a while yet.

[#6]

@bgooley ,

To be honest, I only got this information below and don't 100% understand it because I am not security expert. This is might make more sense to you:

Vulnerability #1: SSL Medium Strength Cipher Suites Supported (SWEET32)

Medium Strength Ciphers (> 64-bit and < 112-bit key or 3DES)

KRB5-DES-CBC3-MD5 Kx=KRB5 Au=KRB5 Enc=3DES-CBC(168) Mac=MD5 

KRB5-DES-CBC3-SHA Kx=KRB5 Au=KRB5 Enc=3DES-CBC(168) Mac=SHA1 

DES-CBC3-SHA Kx=RSA Au=RSA Enc=3DES-CBC(168) Mac=SHA1 

The fields above are :

{OpenSSL ciphername}

Kx={key exchange}

Au={authentication}

Enc={symmetric encryption method}

Mac={message authentication code}

Vulnerability #2: SSL RC4 Cipher Suites Supported (Bar Mitzvah)

List of RC4 cipher suites supported by the remote server :

High Strength Ciphers (>= 112-bit key)

KRB5-RC4-MD5 Kx=KRB5 Au=KRB5 Enc=RC4(128) Mac=MD5

KRB5-RC4-SHA Kx=KRB5 Au=KRB5 Enc=RC4(128) Mac=SHA1

RC4-MD5 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5

RC4-SHA Kx=RSA Au=RSA Enc=RC4(128) Mac=SHA1

The fields above are :

{OpenSSL ciphername}

Kx={key exchange}

Au={authentication}

Enc={symmetric encryption method}

Mac={message authentication code}

I hope this means something to you. 🙂

Thanks,

Brett

[#7]

@BrettM,

This makes me think that the code change did not actually take effect since the cipher list I gave you as an example removes DES and 3DES ciphers.  This worked in my environment so the ciphers that were available before the change were no longer available afterward.

Can you confirm that the agent Nessus is detecting as having vulnerable ciphers has the following code changes in https.py:

  ctx = SSL.Context('sslv23')
  ctx.set_options(m2.SSL_OP_NO_TLSv1 | 0x10000000L | m2.SSL_OP_NO_SSLv2 | m2.SSL_OP_NO_SSLv3)
  ctx.set_cipher_list("HIGH:!DSS:!DH:!ADH:!DES:!3DES:!SHA1:!RC4:!aNULL:!eNULL:!EXPORT:!SSLv2:!SSLv3:!TLSv1")
  ctx.set_session_id_ctx("cm-agent")

I added "!RC4" in there just in case but really "HIGH" should have taken care of that I think.

Also, be sure you restarted the agent after the last change to make sure the changes took place.

[#8]

@bgooley 

Our scan back from Nessus is good on Port 9000. We are not getting any of those aforementioned issues with the previous update from last week you provided. 

I was just letting you know the original message I was getting back about Port 9000. I agree the !HIGH would have removed the RC4 cipher.

We are good, this issue on Port 9000 is remediated.  Thank You very much!

Thanks,

Brett

[#9]

@bgooley 

In CDH 6.3.x, this appears to have changed and the "https.py" file is slightly different now.  It accepts the cipher_list as a configuration item.  The way we secured Port 900 is by doing these

steps:

1) Check to see if RC4 (and other weak ciphers) are open on Port 9000:

openssl s_client -cipher RC4 -connect <server>:9000 -msg

2) Edit the "/etc/cloudera-scm-agent/config.ini" file

3) Under the "[Security]" section of the config.ini file, we added these lines:

# Custom Cipher List to close vulnerabilities for port 9000
cipher_list=HIGH:!DSS:!DH:!ADH:!DES:!3DES:!SHA1:!RC4:!aNULL:!eNULL:!EXPORT:!SSLv2:!SSLv3:!TLSv1

4) Restart the Cloudera CM-Agent:  

sudo service cloudera-scm-agent restart

5) Wait a minute or so and then rerun the OpenSSL command and RC4 (and other weak ciphers, if you test them) are closed:

openssl s_client -cipher RC4 -connect <server>:9000 -msg

It would be great if Cloudera could add this to their documentation on how to add this additional security to the CM Agent.

[#10]

@bgooley 

That was quite a good list of steps I could find after searching a lot for procedures on upgrading the TLS 1.1 to 1.2.

I actually applied these steps on one of our test environment  on CDH 5.13 cluster on centos 6 wihin our organization and submitted for the vulnerability scan and the report has come up with quite a number of ports still have TLS 1.1 and are vulnerable

These are the ports:

11371 -- KTS server

11381-- postgresssql database

50475  External -  Datanode-- dfs.datanode.https.address

13562  Yarn //mapreduce.shuffle.port

9093--kafka

8985 -- solr_https_port

8044 --Yarn,node manager --yarn. nodemanager. webapp.https.address

20550 --hbase.rest.port

19890-- Yarn Job history server, mapreduce. jobhistory. webapp.address

11443 -- Oozie server

9095 --Hbase Thrift server

8889 -- Hue load balancer

60010 -- hbase.master. info.port (http)

7187-- Cloudera manager server (metadataserve/https web UI)

50470 -- dfs.https.address or dfs.namenode.https-address (dfs.https.addressis deprecated (but still works)

14000 -- HttpFS

8481 --Hadoop --dfs.journalnode. https-address

8090 -- yarn. resourcemanager. webapp.https.address

8044 ---yarn. nodemanager. webapp.https.address

60030 -- hbase. regionserver. info.port.

please let us know how we can overcome/resolve this issue.

Thanks

Suresh

.

[#11]

@bgooley 

Hi

That was quite a good list of steps I could find after searching a lot for procedures on upgrading the TLS 1.1 to 1.2.

I actually applied these steps on one of our test environment on CDH 5.13 cluster on centos 6 within our organization and submitted for the vulnerability scan and the report has come up with quite a number of ports still have TLS 1.1 vulnerability.

These are the ports:

11371 -- KTS server

11381-- postgresssql database

50475 External - Datanode-- dfs.datanode.https.address

13562 Yarn //mapreduce.shuffle.port

9093--kafka

8985 -- solr_https_port

8044 --Yarn,node manager --yarn. nodemanager. webapp.https.address

20550 --hbase.rest.port

19890-- Yarn Job history server, mapreduce. jobhistory. webapp.address

11443 -- Oozie server

9095 --Hbase Thrift server

8889 -- Hue load balancer

60010 -- hbase.master. info.port (http)

7187-- Cloudera manager server (metadataserve/https web UI)

50470 -- dfs.https.address or dfs.namenode.https-address (dfs.https.addressis deprecated (but still works)

14000 -- HttpFS

8481 --Hadoop --dfs.journalnode. https-address

8090 -- yarn. resourcemanager. webapp.https.address

8044 ---yarn. nodemanager. webapp.https.address

60030 -- hbase. regionserver. info.port.

please let us know how we can overcome/resolve this issue. Looking forward for your response.

Thanks

Abdul