[SOLVED] java.io.IOException: Login failure for Datanode

[#1]

I can't start one of my DN (rest of them are running)

2016-09-01 16:35:37,489 ERROR datanode.DataNode (DataNode.java:secureMain(2545)) - Exception in secureMain
java.io.IOException: Login failure for dn/datanode@EXAMPLE.COM from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

- File permissions

-r--------. 1 hdfs hadoop 408 Sep  1 15:36 /etc/security/keytabs/dn.service.keytab

- File content

Keytab name: FILE:/etc/security/keytabs/dn.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 09/01/2016 15:36:21 dn/datanode.example.com@EXAMPLE.COM
   1 09/01/2016 15:36:21 dn/datanode.example.com@EXAMPLE.COM
   1 09/01/2016 15:36:21 dn/datanode.example.com@EXAMPLE.COM
   1 09/01/2016 15:36:21 dn/datanode.example.com@EXAMPLE.COM
   1 09/01/2016 15:36:21 dn/datanode.example.com@EXAMPLE.COM

Also, I found that KDC has the principal `dn/datanode.example.com@EXAMPLE.COM` and not `dn/datanode@EXAMPLE.COM`. And this command works:

kinit -kt /etc/security/keytabs/dn.service.keytab dn/datanode.example.com@EXAMPLE.COM

So, why is HDFS using a wrong principal?

Should I regenerate the Kerberos keys from Ambari UI? Thanks in advance.

[#2 · Accepted solution]

I solved the issue: in the file `/etc/hosts` the short hostname is before the long

192.168.1.3 datanode datanode.example.com

I switched the order

192.168.1.3 datanode.example.com datanode