Unable to access HDFS after enabling kerberos

gvh — Mon, 07 Jan 2019 14:16:51 GMT

I have enabled kerberos authentication on my cluster. After that when namenode is not starting due Checksum error. I am using cloudera 6.0.1 on Suse 12 OS hosted on GCP

Failed to start namenode.
org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hdfs/nn.asia-south1-c.c.cdh-cluster2.internal@ASIA-SOUTH1-C.C.CDH-CLUSTER2.INTERNAL from keytab hdfs.keytab javax.security.auth.login.LoginException: Checksum failed
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1843)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1211)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1007)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:313)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:694)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:713)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:950)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
Caused by: javax.security.auth.login.LoginException: Checksum failed
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1921)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1833)
... 9 more
Caused by: KrbException: Checksum failed
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:102)
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:94)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:149)
at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:285)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
... 23 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:408)
at sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:91)
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:100)
... 30 more

Re: Unable to access HDFS after enabling kerberos

Robert Justice — Mon, 07 Jan 2019 21:08:26 GMT

Typically, we see this type of stack trace when the keytab for the service principals do not match the principals in the KDC. Also, occasionally, the enctypes for the KDC do not match up with encryption types in your krb5.conf on the client side, with something like arcfour-hmac missing, but this is less likely the cause.

https://web.mit.edu/kerberos/kfw-4.1/kfw-4.1/kfw-4.1-help/html/encryption_types.htm

Our first suggestion would be to:

Stop all services on the cluster
Regenerating all of the existing principals beneath:
Administration > Security > Kerberos Credentials > Select all > Regenerate Selected
Start all services on the cluster

question Unable to access HDFS after enabling kerberos in Support Questions

Unable to access HDFS after enabling kerberos

Re: Unable to access HDFS after enabling kerberos