CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama

MrBee — Fri, 16 Sep 2022 09:21:52 GMT

Using Cloudera Manager we want to enable Impala on YARN.

We did so by adding the service llama ApplicationMaster and changing the min cores/mem to 0 and enabling cgroups.

We restarted the whole cluster. (HDFS works, Hive on YARN works)

Problem:

(Shell build version: Impala Shell v2.1.0-cdh5 (e48c2b4) built on Tue Dec 16 19:00:35 PST 2014)
[Not connected] > connect data01;
Error connecting: TTransportException, TSocket read 0 bytes
Kerberos ticket found in the credentials cache, retrying the connection with a secure transport.
Connected to data01:21000
Server version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2)
[data01:21000] > use mydb;
Query: use mydb
[data01:21000] > select * from mytable limit 10;
Query: select * from mytable limit 10
ERROR:  com.cloudera.llama.util.LlamaException: AM_CANNOT_REGISTER - cannot register AM 'application_1424245272359_0001' for queue 'root.alexanderbij' : java.lang.reflect.UndeclaredThrowableException, com.cloudera.llama.util.LlamaException: AM_CANNOT_REGISTER - cannot register AM 'application_1424245272359_0001' for queue 'root.alexanderbij' : java.lang.reflect.UndeclaredThrowableException, 	at com.cloudera.llama.am.yarn.YarnRMConnector.register(YarnRMConnector.java:270), 	at com.cloudera.llama.am.cache.CacheRMConnector.register(CacheRMConnector.java:178), 	at com.cloudera.llama.am.impl.NormalizerRMConnector.register(NormalizerRMConnector.java:107), 	at com.cloudera.llama.am.impl.PhasingOutRMConnector.register(PhasingOutRMConnector.java:139), 	at com.cloudera.llama.am.impl.SingleQueueLlamaAM.start(SingleQueueLlamaAM.java:158), 	at com.cloudera.llama.am.impl.ThrottleLlamaAM.start(ThrottleLlamaAM.java:164), 	at com.cloudera.llama.am.impl.MultiQueueLlamaAM.getSingleQueueAMInfo(MultiQueueLlamaAM.java:169), 	at com.cloudera.llama.am.impl.MultiQueueLlamaAM.reserve(MultiQueueLlamaAM.java:286), 	at com.cloudera.llama.am.impl.GangAntiDeadlockLlamaAM.reserve(GangAntiDeadlockLlamaAM.java:205), 	at com.cloudera.llama.am.impl.ExpansionReservationsLlamaAM.reserve(ExpansionReservationsLlamaAM.java:131), 	at com.cloudera.llama.am.impl.APIContractLlamaAM.reserve(APIContractLlamaAM.java:144), 	at com.cloudera.llama.am.LlamaAMServiceImpl.Reserve(LlamaAMServiceImpl.java:132), 	at com.cloudera.llama.am.MetricLlamaAMService.Reserve(MetricLlamaAMService.java:140), 	at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:512), 	at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:497), 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39), 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39), 	at com.cloudera.llama.server.ClientPrincipalTProcessor.process(ClientPrincipalTProcessor.java:47), 	at com.cloudera.llama.server.AuthzTProcessor.process(AuthzTProcessor.java:89), 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206), 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), 	at java.lang.Thread.run(Thread.java:745), Caused by: java.lang.reflect.UndeclaredThrowableException, 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655), 	at com.cloudera.llama.am.yarn.YarnRMConnector.register(YarnRMConnector.java:239), 	... 22 more, Caused by: com.cloudera.llama.util.LlamaException: AM_TIMED_OUT_STARTING_STOPPING - AM 'application_1424245272359_0001' timed out ('30000' ms) in state 'FAILED' transitioning to '[ACCEPTED]' while 'starting', 	at com.cloudera.llama.am.yarn.YarnRMConnector._monitorAppState(YarnRMConnector.java:429), 	at com.cloudera.llama.am.yarn.YarnRMConnector._initYarnApp(YarnRMConnector.java:294), 	at com.cloudera.llama.am.yarn.YarnRMConnector.access$400(YarnRMConnector.java:83), 	at com.cloudera.llama.am.yarn.YarnRMConnector$4.run(YarnRMConnector.java:243), 	at com.cloudera.llama.am.yarn.YarnRMConnector$4.run(YarnRMConnector.java:240), 	at java.security.AccessController.doPrivileged(Native Method), 	at javax.security.auth.Subject.doAs(Subject.java:415), 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642), 	... 23 more
[data01:21000] >

Looking at log in Cloudera Manager (Diagnostics)

PriviledgedActionException as:llama (auth:PROXY) via yarn/master01.mydomain.int@MYDOMAIN (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: yarn/master01.mydomain.int@MYDOMAIN is not allowed to impersonate llama

In the configuration of YARN

Service-Wide > Proxy:

all services including llama have a *.

Looking at the YARN ResourceManager on master01 running process, inspecting the core-site.xml.

I can confirm that these values are applied.

Do you have any clue where the problem might be?

Re: CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama

dice — Wed, 18 Feb 2015 08:18:44 GMT

You may be hitting YARN-2964 which is current limitation of getting
delegation toke over llama.
To prevent the error, you should implement the following configurations.

Configuring YARN for Long-running Applications
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_yarn_long_jobs.html

Re: CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama

MrBee — Wed, 18 Feb 2015 09:13:07 GMT

Hi, thx for your quick response!

This solution did indeed solve the problem.

I was also tried to change the setting in the Clusters > Impala

authorized_proxy_user_config (default: hue=*)

I have changed this to hue=*;yarn=*.

Let me reset this to default and test, without my modifications.

Re: CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama

MrBee — Wed, 18 Feb 2015 09:16:38 GMT

Reset authorized_proxy_user_config to default (hue=*) still works.

question Re: CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama in Archives of Support Questions (Read Only)

CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama

Re: CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama

Re: CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama

Re: CDH 5.3 Impala Llama Kerberos: yarn is not allowed to impersonate llama