Support Questions

Find answers, ask questions, and share your expertise

Exception on HDP cluster(kerberos+Encryption) while starting yarn application


I have HDP2.3 cluster kerberos + HDFS encryption enabled.

While submitting yarn application during token acquisition i am getting following error. java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens( at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens( at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens( at$ at$ at Method) at at at at at at$800( at$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent( at$ at java.util.concurrent.ThreadPoolExecutor.runWorker( at java.util.concurrent.ThreadPoolExecutor$ at Caused by: java.lang.reflect.UndeclaredThrowableException at at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens( ... 16 more Caused by: Authentication failed, status: 403, message: Forbidden at at at at at at at at at at org.apache.hadoop.crypto.key.kms.KMSClientProvider$ at org.apache.hadoop.crypto.key.kms.KMSClientProvider$ at Method) at at ... 17 more

I have ticket generated for user user1 and it is valid. Also i have added configurations such as following for Ranger KMS

hadoop.kms.proxyuser.user1.users = *

hadoop.kms.proxyuser.user1.hosts = *

I had similar issue on client side and it was failing with the same error earlier even on my client machine. But after adding above properties to Ranger KMS at client side calls seems to be through.

But while starting yarn application on cluster side i am facing above mentioned error and i found that from ResourceManager log. User being impersonated to start yarn service is also user1.

Any idea on what else could be missing to make yarn application start? Let me know if more details on the issue is required.


Vishal pls try my suggestions to your other question. The methodology to troubleshoot Ranger/Ranger KMS issue should be the same

View solution in original post


Super Guru

@Vishal Shah

Are you trying give input data to your yarn application from encrypted zone? if so then are you sure that user1 has access to encrypt/decrypt data to/from encrypted zone? have you tried reading/writing file from/to encrypted zone? If not then can you please try this first?

Vishal pls try my suggestions to your other question. The methodology to troubleshoot Ranger/Ranger KMS issue should be the same


I was able to find the issue where in my application internally user being used was not added to kms proxyuser list. After that it started working.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.