Support Questions
Find answers, ask questions, and share your expertise

Error while submitting spark job on a HDFS transparent encryption + kerberos enabled cluster

Highlighted

Error while submitting spark job on a HDFS transparent encryption + kerberos enabled cluster

New Contributor

@Vishal Shah @Ali Bajwa I am trying to run a spark job which simply reads text file from a encrypted zone on the cluster and writes it to a text file in same encyrpted zone. I am using a HDP2.4 cluster. The spark job is submitted by "user1" and I have added following users in the kms-site.xml:

hadoop.kms.proxyuser.HTTP.hosts=* hadoop.kms.proxyuser.HTTP.users=* hadoop.kms.proxyuser.hive.hosts=* hadoop.kms.proxyuser.hive.users=* hadoop.kms.proxyuser.keyadmin.hosts=* hadoop.kms.proxyuser.keyadmin.users=* hadoop.kms.proxyuser.nn.hosts=* hadoop.kms.proxyuser.nn.users=* hadoop.kms.proxyuser.rm.hosts=* hadoop.kms.proxyuser.rm.users=* hadoop.kms.proxyuser.yarn.hosts=* hadoop.kms.proxyuser.yarn.users=* hadoop.kms.proxyuser.user1.users = *

hadoop.kms.proxyuser.user1.hosts = *

I have specified "dfs.encryption.key.provider.uri" in the client-side hdfs-site.xml

I am getting following error:

16/09/01 17:37:48 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1697 for user1 on 12.68.108.49:8020 2016-09-01 17:37:49.117 <Thread-141> INFO: Exception in thread "main" java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 2016-09-01 17:37:49.117 <Thread-141> INFO: at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:794) 2016-09-01 17:37:49.117 <Thread-141> INFO: at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) 2016-09-01 17:37:49.117 <Thread-141> INFO: at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2046) 2016-09-01 17:37:49.117 <Thread-141> INFO: at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:126) 2016-09-01 17:37:49.117 <Thread-141> INFO: at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:123) 2016-09-01 17:37:49.117 <Thread-141> INFO: at scala.collection.immutable.Set$Set1.foreach(Set.scala:74) 2016-09-01 17:37:49.117 <Thread-141> INFO: at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokensForNamenodes(YarnSparkHadoopUtil.scala:123) 2016-09-01 17:37:49.118 <Thread-141> INFO: Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 2016-09-01 17:37:49.118 <Thread-141> INFO: at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) 2016-09-01 17:37:49.118 <Thread-141> INFO: at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) 2016-09-01 17:37:49.118 <Thread-141> INFO: at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) 2016-09-01 17:37:49.118 <Thread-141> INFO: at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) 2016-09-01 17:37:49.121 <Thread-141> INFO: at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284) 2016-09-01 17:37:49.121 <Thread-141> INFO: at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165) 2016-09-01 17:37:49.121 <Thread-141> INFO: at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) 2016-09-01 17:37:49.121 <Thread-141> INFO: at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:348) 2016-09-01 17:37:49.121 <Thread-141> INFO: at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:786) 2016-09-01 17:37:49.122 <Thread-141> INFO: ... 27 more 2016-09-01 17:37:49.122 <Thread-141> INFO: Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 2016-09-01 17:37:49.122 <Thread-141> INFO: at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) 2016-09-01 17:37:49.122 <Thread-141> INFO: at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) 2016-09-01 17:37:49.122 <Thread-141> INFO: at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) 2016-09-01 17:37:49.122 <Thread-141> INFO: at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) 2016-09-01 17:37:49.122 <Thread-141> INFO: at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) 2016-09-01 17:37:49.122 <Thread-141> INFO: at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) 2016-09-01 17:37:49.122 <Thread-141> INFO: at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) 2016-09-01 17:37:49.122 <Thread-141> INFO: at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) 2016-09-01 17:37:49.122 <Thread-141> INFO: at java.security.AccessController.doPrivileged(Native Method) 2016-09-01 17:37:49.122 <Thread-141> INFO: at javax.security.auth.Subject.doAs(Subject.java:422) 2016-09-01 17:37:49.122 <Thread-141> INFO: at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) 2016-09-01 17:37:49.122 <Thread-141> INFO: ... 35 more