Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Solved Go to solution
Highlighted

Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Explorer

I have HDP2.3 cluster kerberos + HDFS encryption enabled.

While submitting yarn application during token acquisition i am getting following error.

java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:888) at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2243) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:663) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:658) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.obtainSystemTokensForUser(DelegationTokenRenewer.java:657) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.requestNewHdfsDelegationToken(DelegationTokenRenewer.java:621) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:483) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:869) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:846) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1672) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:870) ... 16 more Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274) at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:285) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:166) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:875) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:870) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ... 17 more

I have ticket generated for user user1 and it is valid. Also i have added configurations such as following for Ranger KMS

hadoop.kms.proxyuser.user1.users = *

hadoop.kms.proxyuser.user1.hosts = *

I had similar issue on client side and it was failing with the same error earlier even on my client machine. But after adding above properties to Ranger KMS at client side calls seems to be through.

But while starting yarn application on cluster side i am facing above mentioned error and i found that from ResourceManager log. User being impersonated to start yarn service is also user1.

Any idea on what else could be missing to make yarn application start? Let me know if more details on the issue is required.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Vishal pls try my suggestions to your other question. The methodology to troubleshoot Ranger/Ranger KMS issue should be the same

https://community.hortonworks.com/questions/28052/exception-while-executing-insert-query-on-kerberos...

3 REPLIES 3

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Super Guru

@Vishal Shah

Are you trying give input data to your yarn application from encrypted zone? if so then are you sure that user1 has access to encrypt/decrypt data to/from encrypted zone? have you tried reading/writing file from/to encrypted zone? If not then can you please try this first?

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Vishal pls try my suggestions to your other question. The methodology to troubleshoot Ranger/Ranger KMS issue should be the same

https://community.hortonworks.com/questions/28052/exception-while-executing-insert-query-on-kerberos...

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Explorer

I was able to find the issue where in my application internally user being used was not added to kms proxyuser list. After that it started working.