Skip to main content
gvh
gvh
Bronze ⭐️⭐️⭐️
January 7, 2019
Solved

Unable to access HDFS after enabling kerberos

I have enabled kerberos authentication on my cluster. After that when namenode is not starting due Checksum error. I am using cloudera 6.0.1 on Suse 12 OS hosted on GCP


Failed to start namenode.
org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hdfs/nn.asia-south1-c.c.cdh-cluster2.internal@ASIA-SOUTH1-C.C.CDH-CLUSTER2.INTERNAL from keytab hdfs.keytab javax.security.auth.login.LoginException: Checksum failed
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1843)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1211)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1007)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:313)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:694)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:713)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:950)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
Caused by: javax.security.auth.login.LoginException: Checksum failed
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1921)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1833)
... 9 more
Caused by: KrbException: Checksum failed
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:102)
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:94)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:149)
at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:285)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
... 23 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:408)
at sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:91)
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:100)
... 30 more

Robert Justice
Robert JusticeAnswer
Gold ⭐️⭐️
January 7, 2019

Typically, we see this type of stack trace when the keytab for the service principals do not match the principals in the KDC.   Also, occasionally, the enctypes for the KDC do not match up with encryption types in your krb5.conf on the client side, with something like arcfour-hmac missing, but this is less likely the cause.

 

https://web.mit.edu/kerberos/kfw-4.1/kfw-4.1/kfw-4.1-help/html/encryption_types.htm

 

Our first suggestion would be to: 

 

  • Stop all services on the cluster
  • Regenerating all of the existing principals beneath:
  • Administration > Security > Kerberos Credentials > Select all > Regenerate Selected
  • Start all services on the cluster

 

Robert Justice, Technical Resolution Manager
View in thread

1 reply

Robert Justice
Gold ⭐️⭐️
Robert JusticeAnswer
Gold ⭐️⭐️
January 7, 2019

Typically, we see this type of stack trace when the keytab for the service principals do not match the principals in the KDC.   Also, occasionally, the enctypes for the KDC do not match up with encryption types in your krb5.conf on the client side, with something like arcfour-hmac missing, but this is less likely the cause.

 

https://web.mit.edu/kerberos/kfw-4.1/kfw-4.1/kfw-4.1-help/html/encryption_types.htm

 

Our first suggestion would be to: 

 

  • Stop all services on the cluster
  • Regenerating all of the existing principals beneath:
  • Administration > Security > Kerberos Credentials > Select all > Regenerate Selected
  • Start all services on the cluster

 

Translate
Robert Justice, Technical Resolution Manager
    Terms & ConditionsAccessibility statement