Why does Ambari set Headless Keytabs' permissions to 440? Isn't that a security risk?

[#1]

Hey community,

during setting up Kerberos in an HDP 2.3.4 cluster with Ambari, I was wondering the following:

Why does Ambari set the linux permissions for headless keytabs (e.g /etc/security/keytabs/hdfs.headless.keytab) to 440 rather than 400?

I feel that this creates a security risk, as everyone in the hadoop group could impersonate hdfs and thus effectively work as superuser on the cluster. Or is the group "hadoop" supposed to be a super-protected group only used by Hadoop Service Accounts?

Hope you can clear this up for me.

Thanks, Benjamin

edit: It seems to be connected to AMBARI-13695, created by @Robert Levas

[#2]

@Benjamin R

I assume that reason the HDFS headless keytab file's permission 440 rather than 400 is to allow components (what execute under that same group) to use the keytab file in order to create their specific filesystem structures in HDFS during install. However, I am not totally sure.

When adding the feature of enabling Kerberos to Ambari, we intended to ensure similarity with older versions of Ambari where the process was manual - see Ambari 1.7.0 Security Guide - Creating Service Principals and Keytab Files for Hadoop 2.x. As part of that process, the HDFS headless keytab file was chmod-ed to 440.

At some point we want to investigate limiting the exposure of this keytab file (further than we have) by providing a registry so that services can declare what filesystem structure they need in HDFS. Then the HDFS service can create it on behalf of the service. This should allow for the HDFS headless keytab file to be distributed to only the hosts where a NameNode is installed and in turn the permission on the file can be set to 400.

[#3]

Thank you for the insights! This sounds reasonable but still a bit risky.

If the initial creation the file structure is the only reason for the hdfs-keytab to be group-readable, I'm considering to change the permissions to 400 manually and re-change it to 440, when installing new components. This sounds more secure than beeing super-restless on securing the hadoop-group. Any thoughts on that approach?

[#4]

@Benjamin R

That seems reasonable to me. I would be interested to hear if there are any ill effects by doing that. That would help with my proposal for the registry I mentioned.

[#5]

@Robert Levas

I just tried changing the permissions and restarting the cluster. As it turns out, some Ambari services rely on using that keytab with their user on every restart.

In particular: The WebHCat server does not start, failing to execute the following command:

Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-mycluster@REALM.DE;'] {'path': ['/bin'], 'user': 'hcat'}

All other services appear to work correctly.

[#6]

Thanks for the update. I am surprised that WebHCat was the only issue. I see in the scripts why that is happening, but I would imagine other services might follow the same pattern. However, maybe this is a bug in WebHCat and someone might need to investigate why the script does this.

[#7 · Accepted solution]

@Benjamin R

The permissions of the keytabs are not all 440. Only some of them are (hdfs, hbase, ambari-qa, accumulo, and spnego). Those keytabs are used by other services than just the owner for testing connectivity, and other functions on startup. All of the other keytabs are only readable by the service account that owns the file. The group that owns the keytabs is the hadoop group, and should be reserved for service accounts. This will ensure the security of your cluster.