Kerberos integration issue's with hadoop HA

[#1]

I have setup hadoop ha by using cdh 5 and tried to integrate kerberos with it.i could start namenode where kerberos kdc installed successfully.But second namenode startup with an error message..

java.io.IOException: Login failure for hdfs/rhel3.had.com@had.com from keytab /etc/hadoop/conf/hdfs.keytab

2015-02-18 16:24:27,391 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use mycluster to access this namenode/service.
2015-02-18 16:24:28,220 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Login failure for hdfs/rhel3.had.com@had.com from keytab /etc/hadoop/conf/hdfs.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:947)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:560)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:579)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:754)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:738)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1427)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1493)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

kinit works in the name node

[root@rhel3 ~]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 ~]# klist -a
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/18/15 19:47:52  02/19/15 19:47:52  krbtgt/had.com@had.com
        renew until 02/18/15 19:47:52
        Addresses: (none)
[root@rhel3 ~]#

hdfs-site.xml:

<property>
  <name>dfs.block.access.token.enable</name>
  <value>true</value>
</property>

<!-- NameNode security config -->
<property>
  <name>dfs.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
  <name>dfs.namenode.kerberos.principal</name>
  <value>hdfs/rhel3.had.com@had.com</value>
</property>
<property>
  <name>dfs.namenode.kerberos.internal.spnego.principal</name>
  <value>HTTP/rhel3.had.com@had.com</value>
</property>

<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>
<property>
  <name>dfs.web.authentication.kerberos.principal</name>
  <value>HTTP/rhel3.had.com@had.com</value>
</property>

<property>
  <name>dfs.web.authentication.kerberos.keytab</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HTTP keytab -->
</property>

core-site.xml:
<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value> <!-- A value of "simple" would disable security. -->
</property>

<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>

Please let me know how to resolve the issue..

[#2]

It looks the property names are for the primary namenode's. The secondary namenode should have different ones.

See the following guide and the examples: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_sg_secure_hdfs_conf...

<property>
  <name>dfs.secondary.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.principal</name>
  <value>hdfs/_HOST@YOUR-REALM.COM</value>
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  <value>HTTP/_HOST@YOUR-REALM.COM</value>
</property>

[#3]

Thanks for your solution..
But now  I am getting an another error message after the changes done

java.io.IOException: Running in secure mode, but config doesn't have a keytab

[root@rhel3 conf]# tail -15 hdfs-site.xml
<property>
  <name>dfs.secondary.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.principal</name>
  <value>hdfs/rhel3.had.com@had.com</value>
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  <value>HTTP/rhel3.had.com@had.com</value>
</property>


</configuration>
[root@rhel3 conf]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 conf]# klist -a
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/18/15 22:15:54  02/19/15 22:15:54  krbtgt/had.com@had.com
        renew until 02/18/15 22:15:54
        Addresses: (none)
[root@rhel3 conf]# tail -20 /var/log/hadoop-hdfs/hadoop-hdfs-namenode-rhel3.log
2015-02-18 22:13:40,546 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []
2015-02-18 22:13:40,980 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-02-18 22:13:41,127 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-02-18 22:13:41,127 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2015-02-18 22:13:41,130 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is hdfs://mycluster
2015-02-18 22:13:41,130 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use mycluster to access this namenode/service.
2015-02-18 22:13:41,629 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Running in secure mode, but config doesn't have a keytab
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:235)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:560)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:579)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:754)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:738)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1427)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1493)
2015-02-18 22:13:41,636 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2015-02-18 22:13:41,647 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at rhel3.had.com/9.126.40.34
************************************************************/
[root@rhel3 conf]#

[#4]

is there any hint to resolve the issue?..

[#5]

Oops, you've configured NameNode HA so that the properties for Secondary NameNode are not needed. Sorry for my misunderstandings. Please revert back to the previous configurations.

Then please let me know the result of the following commands (I wanted to know the result of "ef" option of klist)

# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
# klist -ef

[#6]

[root@rhel3 ~]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 ~]# klist -ef
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/19/15 17:26:33  02/20/15 17:26:32  krbtgt/had.com@had.com
        renew until 02/19/15 17:26:33, Flags: FRI
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[root@rhel3 ~]#

[#7]

is there anything wrong in the configuration?..

[#8 · Accepted solution]

I have resolved it..it was due to permission of hdfs.keytab

[#9]

What permission did you provide to keytab?