Hacking the KMS javastore

[#1]

Hi,

I've been implemeting a preproduction Cloudera environment in order to test HDFS Data at Rest Encryption. I managed to implement a Java KeyStore KMS succesfully (due to a lack of hardware I'm not able to deploy a KeyTrustee KMS testing).

Testing my environment, I have come across that the root user will be able to gain access to the encripted data in all zones, although it is not configured as the key admin. So, if by chance someone find out the root credentials, they will be able to gather all the encripted data in clear.

These are my findings:

User foo is the owner of the /user/foo directory. This directory is a EncryptedZone created using the "key4foo" key

[foo@management-1 ~]$ hadoop fs -put payrolls /user/foo

[foo@management-1 ~]$ hadoop fs -ls /user/foo

Found 2 items

drwxrwxrwt   - hdfs    foo 0 2016-10-06 07:33 /user/foo/.Trash

-rw-r--r--   2 foo  foo 51 2016-10-06 09:26 /user/foo/payrolls 

User foo can read the payrolls file in clear format. That's the normal behavior.

[foo@management-1 ~]$ hadoop fs -cat /user/foo/payrolls

Charles 1000$

Federico 2000$

Sophie 1111$

Samuel 2020$

But imagine that I'm the sysadmin of the management server and I have root user credentials. So, I would like to check how much money my boss, Charles, is earning.

[root@management-1]$  hadoop fs -cat /user/root/nominas

cat: User [root] is not authorized to perform [DECRYPT_EEK] on key with ACL name [key4esteban]!!

I'm not authorized, normal behavior, so I try to change file permissions.

[root@management-1 ~]# hadoop fs -chmod -R 777 /user/food/

chmod: changing permissions of '/user/esteban': Permission denied. user=root is not the owner of inode=esteban

Not able. So, I just google for how Hadoop KMS is configured and I come across that there's an ACL files which is named "kms-acls.xml".

[root@management-1 ~]# find / -name kms-acls.xml
find: ‘/proc/28573’: No such file or directory
/run/cloudera-scm-agent/process/366-kms-KMS/kms-acls.xml

[root@management-1 ~]# cat /run/cloudera-scm-agent/process/366-kms-KMS/kms-acls.xml | grep foo

    <name>key.acl.key4foo.DECRYPT_EEK</name>

    <value>foo</value>

So, I just have to modify this file and I'd get acess to foo's encrypted zone on the fly.

[root@management-1 ~]# ls -ll /run/cloudera-scm-agent/process/366-kms-KMS/kms-acls.xml

-rw-r----- 1 kms kms 3918 Oct  6 09:21 /run/cloudera-scm-agent/process/366-kms-KMS/kms-acls.xm

[root@management-1 ~]# cat /run/cloudera-scm-agent/process/366-kms-KMS/kms-acls.xml | grep foo

    <name>key.acl.key4foo.DECRYPT_EEK</name>

    <value>foo root</value>

[root@management-1 ~]$ hadoop fs -cat /user/foo/payrolls

Charles 1000$

Federico 2000$

Sophie 1111$

Samuel 2020$

Is this a security hole or am I missing something? Since this kms-acls.xml will run in the management server, will Server KeyTrustee deployment suffer the same security hole?

Any help that you can provide is appreciated.

BR,

Juan.

[#2 · Accepted solution]

Hi Juan,

You're not missing anything. A user with root access to the KMS host, as you suggest, will have the ability to modify KMS and key ACLs. Protecting against a rogue root (or other administrative) user is one of the mosts difficult tasks in IT security. There are several recommended best practices to mitigate this attack vector:

1. KMS hosts should be dedicated so that they can be properly isolated and protected. See the Data at Rest Encryption Reference Architecture.
2. No user should have direct root access to production hosts. Sudo or a similar mechanism should be used.
3. Sudo privileges for specific users and groups should be limited by role.
4. Sudo access to production hosts should be audited by user and possibly also by role. The degree to which actions are audited and the sophistication of the auditing solution will vary and are determined by gauging the level of risk against the necessary degree of protection identified by the IT organization.
5. Proper IT governance calls for the separation of duties. In some organizations this may mean separating the key management, Linux administration and Hadoop administration roles. When these roles are separated, the attack vector you describe requires the collusion of multiple individuals. If separation of duties is not possible or appropriate, the importance of proper auditing increases.

By following these practices you achieve a state in which very few or no individuals have the combination of HDFS file system permissions, Linux permissions and network permissions to achieve this exploit. You also achieve the state in which individuals acting (individually or jointly) to achieve this exploit will have a difficult time doing so without being identified (either during or after the fact, depending on the capabilities of the auditing system).

Use of the Key Trustee Server does not markedly reduce the threat of a rogue root user (particularly a rogue root user with root-level access to hosts performing varied roles). However, use of the Key Trustee Server increases the availability and reliability of the stored encryption zone keys while decreasing the vulnerabilty of the stored keys to unauthorized access, modification or destruction. The availability and reliability increase is provided by the ability to deploy a multi-node KMS/KT topology (see the reference architecture linked above). The decrease in vulnerability of the stored keys to unauthorized access is provided by allowing for the keys to be stored in a separate network zone from the Hadoop cluster (access to that network zone can be more strictly limited than access to the network zone in which the cluster operates). This vulnerability can be even further decreased by using Key HSM (and your enterprise HSM deployment) along with Key Trustee Server.

Anthony

[#3]

Thanks for your answer Anthony! It really helps me.