Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to Submit a Spark Job in a Kerberized HDP Cluster

Highlighted

How to Submit a Spark Job in a Kerberized HDP Cluster

New Contributor

Hi Community,

 

We have a hadoop cluster that is enabled with kerberos authentication and ranger.

after kerberizing the cluster I have created an encryption zone for hdfs which is working absolutely fine from the command line for both hdfs and hbase.

 

now we need to do read the data from hdfs and write in hbase using a spark job submit using java.

 

we are using same principals which we have tested in command line but this same principal does not working with spark job submit and we are facing the issue for cheksum failed for long time now..

 

can someone guide what is the correct way to authenticate spark job in a kerberized cluster with hdfs and hbase for a single user principal ?

 

we expect following to summarize

 

  1. login thru user "abc" in unix
  2. connect with abc/host@REALM.COM principal
  3. read data from hdfs
  4. use the same principal  abc/host@REALM.COM to write in hbase

 

all this have to be done programmatically in java. if anyone has does it , please share how to do it.

 

currently our code is giving error of checksum failed at step 4..

 

Caused by: javax.security.auth.login.LoginException: Checksum failed

        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)

        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)

        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)

        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)

        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)

        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)

        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1340)

        ... 12 more

Caused by: KrbException: Checksum failed

        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102)

        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94)

        at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)

        at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:149)

        at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)

        at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:285)

        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)

        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)

        ... 25 more

Caused by: java.security.GeneralSecurityException: Checksum failed

        at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)

        at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)

        at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76)

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here