Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Kerberos integration issue's with hadoop HA

avatar
author-rank Bobin
Explorer

Created on ‎02-18-2015 12:54 AM - edited ‎09-16-2022 02:21 AM

I have setup hadoop ha by using cdh 5 and tried to integrate kerberos with it.i could start namenode where kerberos kdc installed successfully.But second namenode startup with an error message..

 

java.io.IOException: Login failure for hdfs/rhel3.had.com@had.com from keytab /etc/hadoop/conf/hdfs.keytab

 

2015-02-18 16:24:27,391 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use mycluster to access this namenode/service.
2015-02-18 16:24:28,220 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Login failure for hdfs/rhel3.had.com@had.com from keytab /etc/hadoop/conf/hdfs.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:947)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:242)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:560)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:579)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:754)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:738)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1427)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1493)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

 

kinit works in the name node

 

[root@rhel3 ~]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 ~]# klist -a
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/18/15 19:47:52  02/19/15 19:47:52  krbtgt/had.com@had.com
        renew until 02/18/15 19:47:52
        Addresses: (none)
[root@rhel3 ~]#

 

hdfs-site.xml:

<property>
  <name>dfs.block.access.token.enable</name>
  <value>true</value>
</property>

<!-- NameNode security config -->
<property>
  <name>dfs.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
  <name>dfs.namenode.kerberos.principal</name>
  <value>hdfs/rhel3.had.com@had.com</value>
</property>
<property>
  <name>dfs.namenode.kerberos.internal.spnego.principal</name>
  <value>HTTP/rhel3.had.com@had.com</value>
</property>

<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>
<property>
  <name>dfs.web.authentication.kerberos.principal</name>
  <value>HTTP/rhel3.had.com@had.com</value>
</property>

<property>
  <name>dfs.web.authentication.kerberos.keytab</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HTTP keytab -->
</property>

core-site.xml:
<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value> <!-- A value of "simple" would disable security. -->
</property>

<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>

Please let me know how to resolve the issue..


 

38,618 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank Bobin
Explorer

Created ‎02-19-2015 10:57 PM

I have resolved it..it was due to permission of hdfs.keytab

View solution in original post

39,372 Views
0 Kudos
1
8 REPLIES 8

avatar
author-rank dice author-rank
Expert Contributor

Created ‎02-18-2015 01:56 AM

It looks the property names are for the primary namenode's. The secondary namenode should have different ones.

See the following guide and the examples: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_sg_secure_hdfs_conf...

 

 

<property>
  <name>dfs.secondary.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.principal</name>
  <value>hdfs/_HOST@YOUR-REALM.COM</value>
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  <value>HTTP/_HOST@YOUR-REALM.COM</value>
</property>

 

 

38,616 Views
0 Kudos

avatar
author-rank Bobin
Explorer

Created ‎02-18-2015 03:20 AM

Thanks for your solution..
But now  I am getting an another error message after the changes done

java.io.IOException: Running in secure mode, but config doesn't have a keytab

[root@rhel3 conf]# tail -15 hdfs-site.xml
<property>
  <name>dfs.secondary.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value> <!-- path to the HDFS keytab -->
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.principal</name>
  <value>hdfs/rhel3.had.com@had.com</value>
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  <value>HTTP/rhel3.had.com@had.com</value>
</property>


</configuration>
[root@rhel3 conf]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 conf]# klist -a
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/18/15 22:15:54  02/19/15 22:15:54  krbtgt/had.com@had.com
        renew until 02/18/15 22:15:54
        Addresses: (none)
[root@rhel3 conf]# tail -20 /var/log/hadoop-hdfs/hadoop-hdfs-namenode-rhel3.log
2015-02-18 22:13:40,546 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []
2015-02-18 22:13:40,980 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-02-18 22:13:41,127 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-02-18 22:13:41,127 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2015-02-18 22:13:41,130 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is hdfs://mycluster
2015-02-18 22:13:41,130 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use mycluster to access this namenode/service.
2015-02-18 22:13:41,629 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.IOException: Running in secure mode, but config doesn't have a keytab
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:235)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:560)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:579)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:754)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:738)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1427)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1493)
2015-02-18 22:13:41,636 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2015-02-18 22:13:41,647 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at rhel3.had.com/9.126.40.34
************************************************************/
[root@rhel3 conf]#

38,613 Views
0 Kudos

avatar
author-rank Bobin
Explorer

Created ‎02-18-2015 07:48 PM

is there any hint to resolve the issue?..

38,590 Views
0 Kudos

avatar
author-rank dice author-rank
Expert Contributor

Created ‎02-18-2015 09:40 PM

Oops, you've configured NameNode HA so that the properties for Secondary NameNode are not needed. Sorry for my misunderstandings. Please revert back to the previous configurations.

Then please let me know the result of the following commands (I wanted to know the result of "ef" option of klist)

 

# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
# klist -ef

38,586 Views
0 Kudos

avatar
author-rank Bobin
Explorer

Created ‎02-18-2015 10:28 PM

[root@rhel3 ~]# kinit -kt /etc/hadoop/conf/hdfs.keytab hdfs/rhel3.had.com
[root@rhel3 ~]# klist -ef
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs/rhel3.had.com@had.com

Valid starting     Expires            Service principal
02/19/15 17:26:33  02/20/15 17:26:32  krbtgt/had.com@had.com
        renew until 02/19/15 17:26:33, Flags: FRI
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[root@rhel3 ~]#

38,584 Views
0 Kudos

avatar
author-rank Bobin
Explorer

Created ‎02-19-2015 05:00 AM

is there anything wrong in the configuration?..

38,559 Views
0 Kudos

avatar
author-rank Bobin
Explorer

Created ‎02-19-2015 10:57 PM

I have resolved it..it was due to permission of hdfs.keytab

39,373 Views
0 Kudos
1

avatar
author-rank Badal
New Contributor

Created ‎06-25-2018 07:13 AM

What permission did you provide to keytab?
20,733 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements