question Re: Kerberos integration issue's with hadoop HA in Archives of Support Questions (Read Only)

Kerberos integration issue's with hadoop HA

Bobin — Fri, 16 Sep 2022 09:21:55 GMT

I have setup hadoop ha by using cdh 5 and tried to integrate kerberos with it.i could start namenode where kerberos kdc installed successfully.But second namenode startup with an error message..

java.io.IOException: Login failure for hdfs/rhel3.had.com@had.com from keytab /etc/hadoop/conf/hdfs.keytab

2015-02-18 16:24:27,391 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use mycluster to access this namenode/service.
2015-02-18 16:24:28,220 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Login failure for hdfs/rhel3.had.com@had.com from keytab /etc/hadoop/conf/hdfs.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:947)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:560)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:579)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:754)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:738)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1427)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1493)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

kinit works in the name node

[root@rhel3 ~]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 ~]# klist -a
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/18/15 19:47:52 02/19/15 19:47:52 krbtgt/had.com@had.com
        renew until 02/18/15 19:47:52
        Addresses: (none)
[root@rhel3 ~]#

hdfs-site.xml:

<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>


<property>
<name>dfs.namenode.keytab.file</name>
<value>/etc/hadoop/conf/hdfs.keytab</value> 
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/rhel3.had.com@had.com</value>
</property>
<property>
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/rhel3.had.com@had.com</value>
</property>

<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/rhel3.had.com@had.com</value>
</property>

<property>
<name>dfs.web.authentication.kerberos.keytab</name>
<value>/etc/hadoop/conf/hdfs.keytab</value> 
</property>

core-site.xml:
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value> 
</property>

<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>

Please let me know how to resolve the issue..

Re: Kerberos integration issue's with hadoop HA

dice — Wed, 18 Feb 2015 09:56:42 GMT

It looks the property names are for the primary namenode's. The secondary namenode should have different ones.

See the following guide and the examples: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_sg_secure_hdfs_config.html

<property>
  <name>dfs.secondary.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.principal</name>
  <value>hdfs/_HOST@YOUR-REALM.COM</value>
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  <value>HTTP/_HOST@YOUR-REALM.COM</value>
</property>

Re: Kerberos integration issue's with hadoop HA

Bobin — Wed, 18 Feb 2015 11:20:29 GMT

Thanks for your solution..
But now I am getting an another error message after the changes done

java.io.IOException: Running in secure mode, but config doesn't have a keytab

[root@rhel3 conf]# tail -15 hdfs-site.xml
<property>
<name>dfs.secondary.namenode.keytab.file</name>
<value>/etc/hadoop/conf/hdfs.keytab</value> 
</property>
<property>
<name>dfs.secondary.namenode.kerberos.principal</name>
<value>hdfs/rhel3.had.com@had.com</value>
</property>
<property>
<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/rhel3.had.com@had.com</value>
</property>

</configuration>
[root@rhel3 conf]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 conf]# klist -a
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/18/15 22:15:54 02/19/15 22:15:54 krbtgt/had.com@had.com
        renew until 02/18/15 22:15:54
        Addresses: (none)
[root@rhel3 conf]# tail -20 /var/log/hadoop-hdfs/hadoop-hdfs-namenode-rhel3.log
2015-02-18 22:13:40,546 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []
2015-02-18 22:13:40,980 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-02-18 22:13:41,127 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-02-18 22:13:41,127 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2015-02-18 22:13:41,130 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is hdfs://mycluster
2015-02-18 22:13:41,130 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use mycluster to access this namenode/service.
2015-02-18 22:13:41,629 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Running in secure mode, but config doesn't have a keytab
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:235)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:560)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:579)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:754)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:738)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1427)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1493)
2015-02-18 22:13:41,636 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2015-02-18 22:13:41,647 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at rhel3.had.com/9.126.40.34
************************************************************/
[root@rhel3 conf]#

Re: Kerberos integration issue's with hadoop HA

Bobin — Thu, 19 Feb 2015 03:48:51 GMT

is there any hint to resolve the issue?..

Re: Kerberos integration issue's with hadoop HA

dice — Thu, 19 Feb 2015 05:40:33 GMT

Oops, you've configured NameNode HA so that the properties for Secondary NameNode are not needed. Sorry for my misunderstandings. Please revert back to the previous configurations.

Then please let me know the result of the following commands (I wanted to know the result of "ef" option of klist)

# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
# klist -ef

Re: Kerberos integration issue's with hadoop HA

Bobin — Thu, 19 Feb 2015 06:28:58 GMT

[root@rhel3 ~]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 ~]# klist -ef
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/19/15 17:26:33 02/20/15 17:26:32 krbtgt/had.com@had.com
        renew until 02/19/15 17:26:33, Flags: FRI
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[root@rhel3 ~]#

Re: Kerberos integration issue's with hadoop HA

Bobin — Thu, 19 Feb 2015 13:00:09 GMT

is there anything wrong in the configuration?..

Re: Kerberos integration issue's with hadoop HA

Bobin — Fri, 20 Feb 2015 06:57:32 GMT

I have resolved it..it was due to permission of hdfs.keytab

Re: Kerberos integration issue's with hadoop HA

Badal — Mon, 25 Jun 2018 14:13:07 GMT

What permission did you provide to keytab?