Using Cloudera Manager we want to enable Impala on YARN.
We did so by adding the service llama ApplicationMaster and changing the min cores/mem to 0 and enabling cgroups.
We restarted the whole cluster. (HDFS works, Hive on YARN works)
Problem:
(Shell build version: Impala Shell v2.1.0-cdh5 (e48c2b4) built on Tue Dec 16 19:00:35 PST 2014)
[Not connected] > connect data01;
Error connecting: TTransportException, TSocket read 0 bytes
Kerberos ticket found in the credentials cache, retrying the connection with a secure transport.
Connected to data01:21000
Server version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2)
[data01:21000] > use mydb;
Query: use mydb
[data01:21000] > select * from mytable limit 10;
Query: select * from mytable limit 10
ERROR: com.cloudera.llama.util.LlamaException: AM_CANNOT_REGISTER - cannot register AM 'application_1424245272359_0001' for queue 'root.alexanderbij' : java.lang.reflect.UndeclaredThrowableException, com.cloudera.llama.util.LlamaException: AM_CANNOT_REGISTER - cannot register AM 'application_1424245272359_0001' for queue 'root.alexanderbij' : java.lang.reflect.UndeclaredThrowableException, at com.cloudera.llama.am.yarn.YarnRMConnector.register(YarnRMConnector.java:270), at com.cloudera.llama.am.cache.CacheRMConnector.register(CacheRMConnector.java:178), at com.cloudera.llama.am.impl.NormalizerRMConnector.register(NormalizerRMConnector.java:107), at com.cloudera.llama.am.impl.PhasingOutRMConnector.register(PhasingOutRMConnector.java:139), at com.cloudera.llama.am.impl.SingleQueueLlamaAM.start(SingleQueueLlamaAM.java:158), at com.cloudera.llama.am.impl.ThrottleLlamaAM.start(ThrottleLlamaAM.java:164), at com.cloudera.llama.am.impl.MultiQueueLlamaAM.getSingleQueueAMInfo(MultiQueueLlamaAM.java:169), at com.cloudera.llama.am.impl.MultiQueueLlamaAM.reserve(MultiQueueLlamaAM.java:286), at com.cloudera.llama.am.impl.GangAntiDeadlockLlamaAM.reserve(GangAntiDeadlockLlamaAM.java:205), at com.cloudera.llama.am.impl.ExpansionReservationsLlamaAM.reserve(ExpansionReservationsLlamaAM.java:131), at com.cloudera.llama.am.impl.APIContractLlamaAM.reserve(APIContractLlamaAM.java:144), at com.cloudera.llama.am.LlamaAMServiceImpl.Reserve(LlamaAMServiceImpl.java:132), at com.cloudera.llama.am.MetricLlamaAMService.Reserve(MetricLlamaAMService.java:140), at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:512), at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:497), at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39), at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39), at com.cloudera.llama.server.ClientPrincipalTProcessor.process(ClientPrincipalTProcessor.java:47), at com.cloudera.llama.server.AuthzTProcessor.process(AuthzTProcessor.java:89), at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), at java.lang.Thread.run(Thread.java:745), Caused by: java.lang.reflect.UndeclaredThrowableException, at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655), at com.cloudera.llama.am.yarn.YarnRMConnector.register(YarnRMConnector.java:239), ... 22 more, Caused by: com.cloudera.llama.util.LlamaException: AM_TIMED_OUT_STARTING_STOPPING - AM 'application_1424245272359_0001' timed out ('30000' ms) in state 'FAILED' transitioning to '[ACCEPTED]' while 'starting', at com.cloudera.llama.am.yarn.YarnRMConnector._monitorAppState(YarnRMConnector.java:429), at com.cloudera.llama.am.yarn.YarnRMConnector._initYarnApp(YarnRMConnector.java:294), at com.cloudera.llama.am.yarn.YarnRMConnector.access$400(YarnRMConnector.java:83), at com.cloudera.llama.am.yarn.YarnRMConnector$4.run(YarnRMConnector.java:243), at com.cloudera.llama.am.yarn.YarnRMConnector$4.run(YarnRMConnector.java:240), at java.security.AccessController.doPrivileged(Native Method), at javax.security.auth.Subject.doAs(Subject.java:415), at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642), ... 23 more
[data01:21000] > Looking at log in Cloudera Manager (Diagnostics)
PriviledgedActionException as:llama (auth:PROXY) via yarn/master01.mydomain.int@MYDOMAIN (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: yarn/master01.mydomain.int@MYDOMAIN is not allowed to impersonate llama
In the configuration of YARN
Service-Wide > Proxy:
all services including llama have a *.
Looking at the YARN ResourceManager on master01 running process, inspecting the core-site.xml.
I can confirm that these values are applied.
Do you have any clue where the problem might be?
