Exception on HDP cluster(kerberos+Encryption) while starting yarn application

vmshah — Sun, 24 Apr 2016 00:12:57 GMT

I have HDP2.3 cluster kerberos + HDFS encryption enabled.

While submitting yarn application during token acquisition i am getting following error.

java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:888) at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86) at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2243) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:663) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:658) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.obtainSystemTokensForUser(DelegationTokenRenewer.java:657) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.requestNewHdfsDelegationToken(DelegationTokenRenewer.java:621) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:483) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:869) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:846) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1672) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:870) ... 16 more Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274) at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:285) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:166) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:875) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:870) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ... 17 more

I have ticket generated for user user1 and it is valid. Also i have added configurations such as following for Ranger KMS

hadoop.kms.proxyuser.user1.users = *

hadoop.kms.proxyuser.user1.hosts = *

I had similar issue on client side and it was failing with the same error earlier even on my client machine. But after adding above properties to Ranger KMS at client side calls seems to be through.

But while starting yarn application on cluster side i am facing above mentioned error and i found that from ResourceManager log. User being impersonated to start yarn service is also user1.

Any idea on what else could be missing to make yarn application start? Let me know if more details on the issue is required.

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

KuldeepK — Mon, 25 Apr 2016 12:41:52 GMT

@Vishal Shah

Are you trying give input data to your yarn application from encrypted zone? if so then are you sure that user1 has access to encrypt/decrypt data to/from encrypted zone? have you tried reading/writing file from/to encrypted zone? If not then can you please try this first?

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

abajwa — Tue, 26 Apr 2016 04:39:21 GMT

Vishal pls try my suggestions to your other question. The methodology to troubleshoot Ranger/Ranger KMS issue should be the same

https://community.hortonworks.com/questions/28052/exception-while-executing-insert-query-on-kerberos.html

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

vmshah — Wed, 27 Apr 2016 21:51:58 GMT

I was able to find the issue where in my application internally user being used was not added to kms proxyuser list. After that it started working.

question Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application in Archives of Support Questions (Read Only)

Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application

Re: Exception on HDP cluster(kerberos+Encryption) while starting yarn application