Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama

avatar
Expert Contributor

Using Cloudera Manager we want to enable Impala on YARN.

We did so by adding the service llama ApplicationMaster and changing the min cores/mem to 0 and enabling cgroups.

We restarted the whole cluster. (HDFS works, Hive on YARN works)

 

Problem:

 

(Shell build version: Impala Shell v2.1.0-cdh5 (e48c2b4) built on Tue Dec 16 19:00:35 PST 2014)
[Not connected] > connect data01;
Error connecting: TTransportException, TSocket read 0 bytes
Kerberos ticket found in the credentials cache, retrying the connection with a secure transport.
Connected to data01:21000
Server version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2)
[data01:21000] > use mydb;
Query: use mydb
[data01:21000] > select * from mytable limit 10;
Query: select * from mytable limit 10
ERROR:  com.cloudera.llama.util.LlamaException: AM_CANNOT_REGISTER - cannot register AM 'application_1424245272359_0001' for queue 'root.alexanderbij' : java.lang.reflect.UndeclaredThrowableException, com.cloudera.llama.util.LlamaException: AM_CANNOT_REGISTER - cannot register AM 'application_1424245272359_0001' for queue 'root.alexanderbij' : java.lang.reflect.UndeclaredThrowableException, 	at com.cloudera.llama.am.yarn.YarnRMConnector.register(YarnRMConnector.java:270), 	at com.cloudera.llama.am.cache.CacheRMConnector.register(CacheRMConnector.java:178), 	at com.cloudera.llama.am.impl.NormalizerRMConnector.register(NormalizerRMConnector.java:107), 	at com.cloudera.llama.am.impl.PhasingOutRMConnector.register(PhasingOutRMConnector.java:139), 	at com.cloudera.llama.am.impl.SingleQueueLlamaAM.start(SingleQueueLlamaAM.java:158), 	at com.cloudera.llama.am.impl.ThrottleLlamaAM.start(ThrottleLlamaAM.java:164), 	at com.cloudera.llama.am.impl.MultiQueueLlamaAM.getSingleQueueAMInfo(MultiQueueLlamaAM.java:169), 	at com.cloudera.llama.am.impl.MultiQueueLlamaAM.reserve(MultiQueueLlamaAM.java:286), 	at com.cloudera.llama.am.impl.GangAntiDeadlockLlamaAM.reserve(GangAntiDeadlockLlamaAM.java:205), 	at com.cloudera.llama.am.impl.ExpansionReservationsLlamaAM.reserve(ExpansionReservationsLlamaAM.java:131), 	at com.cloudera.llama.am.impl.APIContractLlamaAM.reserve(APIContractLlamaAM.java:144), 	at com.cloudera.llama.am.LlamaAMServiceImpl.Reserve(LlamaAMServiceImpl.java:132), 	at com.cloudera.llama.am.MetricLlamaAMService.Reserve(MetricLlamaAMService.java:140), 	at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:512), 	at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:497), 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39), 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39), 	at com.cloudera.llama.server.ClientPrincipalTProcessor.process(ClientPrincipalTProcessor.java:47), 	at com.cloudera.llama.server.AuthzTProcessor.process(AuthzTProcessor.java:89), 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206), 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), 	at java.lang.Thread.run(Thread.java:745), Caused by: java.lang.reflect.UndeclaredThrowableException, 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655), 	at com.cloudera.llama.am.yarn.YarnRMConnector.register(YarnRMConnector.java:239), 	... 22 more, Caused by: com.cloudera.llama.util.LlamaException: AM_TIMED_OUT_STARTING_STOPPING - AM 'application_1424245272359_0001' timed out ('30000' ms) in state 'FAILED' transitioning to '[ACCEPTED]' while 'starting', 	at com.cloudera.llama.am.yarn.YarnRMConnector._monitorAppState(YarnRMConnector.java:429), 	at com.cloudera.llama.am.yarn.YarnRMConnector._initYarnApp(YarnRMConnector.java:294), 	at com.cloudera.llama.am.yarn.YarnRMConnector.access$400(YarnRMConnector.java:83), 	at com.cloudera.llama.am.yarn.YarnRMConnector$4.run(YarnRMConnector.java:243), 	at com.cloudera.llama.am.yarn.YarnRMConnector$4.run(YarnRMConnector.java:240), 	at java.security.AccessController.doPrivileged(Native Method), 	at javax.security.auth.Subject.doAs(Subject.java:415), 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642), 	... 23 more
[data01:21000] >

 Looking at log in Cloudera Manager (Diagnostics)

 

PriviledgedActionException as:llama (auth:PROXY) via yarn/master01.mydomain.int@MYDOMAIN (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: yarn/master01.mydomain.int@MYDOMAIN is not allowed to impersonate llama

In the configuration of YARN 

Service-Wide > Proxy: 

all services including llama have a *.

 

Looking at the YARN ResourceManager on master01 running process, inspecting the core-site.xml.

I can confirm that these values are applied.

 

Do you have any clue where the problem might be?

 

 

 

1 ACCEPTED SOLUTION

avatar
Expert Contributor
You may be hitting YARN-2964 which is current limitation of getting
delegation toke over llama.
To prevent the error, you should implement the following configurations.

Configuring YARN for Long-running Applications
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_yarn_long_jobs.ht...


View solution in original post

3 REPLIES 3

avatar
Expert Contributor
You may be hitting YARN-2964 which is current limitation of getting
delegation toke over llama.
To prevent the error, you should implement the following configurations.

Configuring YARN for Long-running Applications
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_yarn_long_jobs.ht...


avatar
Expert Contributor

Hi, thx for your quick response!

 

This solution did indeed solve the problem.

 

I was also tried to change the setting in the Clusters > Impala

authorized_proxy_user_config  (default:  hue=*)

I have changed this to hue=*;yarn=*.

 

Let me reset this to default and test, without my modifications.

avatar
Expert Contributor

Reset authorized_proxy_user_config to default (hue=*) still works.