Kerberos Implementation - Datanodes wont start

[#1]

I’m not a Kerberos wizard, so I’m on a bit of a learning curve. I’ve followed all of the Kerberos instructions in the HDP 2.1 documentation and run into an issue where my datanodes won’t start (3 node cluster). If I roll back all of the xml files to non-kerberos versions, I can start everything from the command line. When I shut down the cluster and roll in the kerberos versions of the xml files, I’m able to start the namenode, but all of the datanodes refuse to start and the only clue I have is as follows;

************************************************************/2014-07-24 11:04:22,181 INFO datanode.DataNode (SignalLogger.java:register(91)) - registered UNIX signal handlers for [TERM, HUP, INT]2014-07-24 11:04:22,399 WARN common.Util (Util.java:stringAsURI(56)) - Path /opt/hadoop/hdfs/dn should be specified as a URI in configuration files. Please update hdfs configuration.2014-07-24 11:04:23,055 INFO security.UserGroupInformation (UserGroupInformation.java:loginUserFromKeytab(894)) - Login successful for user dn/abc0123.xy.local@XYZ.COM using keytab file /etc/security/keytabs/dn.service.keytab2014-07-24 11:04:23,210 INFO impl.MetricsConfig (MetricsConfig.java:loadFirst(111)) - loaded properties from hadoop-metrics2.properties2014-07-24 11:04:23,274 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(344)) - Scheduled snapshot period at 60 second(s).2014-07-24 11:04:23,274 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:start(183)) - DataNode metrics system started2014-07-24 11:04:23,279 INFO datanode.DataNode (DataNode.java:<init>(269)) - File descriptor passing is enabled.2014-07-24 11:04:23,283 INFO datanode.DataNode (DataNode.java:<init>(280)) - Configured hostname is cvm0932.dg.local2014-07-24 11:04:23,284 FATAL datanode.DataNode (DataNode.java:secureMain(2002)) - Exception in secureMainjava.lang.RuntimeException: Cannot start secure cluster without privileged resources.at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:700)at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:281)at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1885)at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1772)at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1819)at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1995)at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2019)2014-07-24 11:04:23,287 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 12014-07-24 11:04:23,289 INFO datanode.DataNode (StringUtils.java:run(640)) - SHUTDOWN_MSG:/********************************** 

[#2]

You start the datanodes as root user in secure setup. Did you do that?

[#3]

What HDP version are you using? Did you use the HDP 2.1 documentation to install Kerberos on HDP 2.3? We don't have the manual installation yet for HDP 2.3 for Kerberos but Ambari can be used to Kerberized HDP 2.3 cluster.

[#4 · Accepted solution]

Likely, one of the following issues -

1) If you enabled kerberos manually, an error was made. You should check if JSVC package is installed.

2) If you installed using Ambari to enable kerberos, then make sure that you are starting datanodes as root i.e. make sure Ambari Server is running as root.

[#5]

Why are the dates in the log from 7/24/2014? Is this an old issue that hasn't been solved and you are reposting it, or is your clock incorrect?

If your clock is incorrect, than you will have Kerberos issues since time is a big factor in determining the validity of the credentials. The clocks on the hosts need to be within 5 minutes of the host that contains the KDC, else bad things will happen.

If this is an old issue and you are using HDP 2.1, then I assume you are using Ambari 1.6.x. In this version of Ambari, you must have set up Kerberos manually. Since there is a lot of room for error, you should go back and make sure you didn't miss a step or incorrect create a keytab file. Unless you create the keytab file for a particular principal using kadmin.local, the password for the account will get regenerated. This will cause issues if you create multiple keytab files for the same principal - the 2nd time you generate a keytab file, the 1st keytab file will become obsolete; the 3rd time you generate a keytab file, the 2nd keytab file will become obsolete, etc...

Also, make sure all of the configs were set properly. By incorrectly setting a principal name or keytab file location, one or more services will fail to authenticate.

Finally, check the ACLs on the keytab files to make sure that the relevant service(s) can read them. If a service is running as the local hdfs user, but the keytab file is only readable by root, than the service cannot read the keytab file and authentication will fail.