Reply
gvh
New Contributor
Posts: 1
Registered: ‎01-07-2019

Unable to access HDFS after enabling kerberos

[ Edited ]

I have enabled kerberos authentication on my cluster. After that when namenode is not starting due Checksum error. I am using cloudera 6.0.1 on Suse 12 OS hosted on GCP


Failed to start namenode.
org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hdfs/nn.asia-south1-c.c.cdh-cluster2.internal@ASIA-SOUTH1-C.C.CDH-CLUSTER2.INTERNAL from keytab hdfs.keytab javax.security.auth.login.LoginException: Checksum failed
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1843)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1211)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1007)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:313)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:694)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:713)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:950)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
Caused by: javax.security.auth.login.LoginException: Checksum failed
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1921)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1833)
... 9 more
Caused by: KrbException: Checksum failed
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:102)
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:94)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:149)
at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:285)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
... 23 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:408)
at sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:91)
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:100)
... 30 more

Posts: 132
Kudos: 3
Solutions: 3
Registered: ‎08-14-2013

Re: Unable to access HDFS after enabling kerberos

Typically, we see this type of stack trace when the keytab for the service principals do not match the principals in the KDC.   Also, occasionally, the enctypes for the KDC do not match up with encryption types in your krb5.conf on the client side, with something like arcfour-hmac missing, but this is less likely the cause.

 

https://web.mit.edu/kerberos/kfw-4.1/kfw-4.1/kfw-4.1-help/html/encryption_types.htm

 

Our first suggestion would be to: 

 

  • Stop all services on the cluster
  • Regenerating all of the existing principals beneath:
  • Administration > Security > Kerberos Credentials > Select all > Regenerate Selected
  • Start all services on the cluster

 



Robert Justice, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Announcements