Member since
06-15-2015
13
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7904 | 07-01-2015 01:25 AM |
01-05-2018
10:30 PM
1 Kudo
It's worth reading this https://community.hortonworks.com/questions/146229/hive-llap-has-conflict-with-hive-sql-standadard-au.html
... View more
12-07-2017
12:15 PM
Thank you for your response. Indeed, 129s is less than 2 x 78s. I've looked into cluster utilisation and your point about the cluster being fully utilised when executing the single query is well made.
... View more
12-05-2017
07:08 PM
Hi anarasimham, Thanks for your response. To answer your questions: # concurrent queries: 3 Memory per node: 62G Number of nodes: 3 Memory allocated to LLAP: "Memory per Daemon" is 18432; llap0 shows 56320MB allocated in YARN ResourceManager UI Please let me know if there is anything else you would like to know. Thanks, Martin
... View more
12-05-2017
01:03 PM
I'm investigating running concurrent queries in LLAP (HDP 2.6.1).
When running more than 1 query I am seeing non-zero "Killed Task Attempts" in the DAG page in Ambari (Tez View -> DAG ID link). In some cases the number of "Killed Task Attempts" is into the hundreds. The logs show messages of the form: 2017-12-05T12:13:42,003 INFO [IPC Server handler 2 on 33976 (1512469369970_0013_2_00_000022_1)] impl.TaskRunnerCallable: Kill task requested for id=attempt_1512469369970_0013_2_00_000022_1, taskRunnerSetup=false
2017-12-05T12:13:42,003 INFO [IPC Server handler 2 on 33976 (1512469369970_0013_2_00_000022_1)] impl.ContainerRunnerImpl: SubmissionState for attempt_1512469369970_0013_2_00_000022_1 : REJECTED
2017-12-05T12:13:42,009 INFO [IPC Server handler 4 on 33976 (1512469369970_0013_2_00_000004_1)] impl.TaskExecutorService: wait queue full, size=10. numSlotsAvailable=0, runningFragmentCount=3. attempt_1512469369970_0013_2_00_000004_1 not added
In some cases REJECTED is EVICTED_OTHER.
The query in isolation executed in 78s. When two executions of the same query run concurrently the execution times were 125s and 129s. As the number of concurrent executions is increased the mean elapsed time continues to rise. There is variation in elapsed time between runs whenever there is more than 1 query execution at a time. There is also a correlation between the elapsed time and the number of "Killed Task Attempts". The following is from executing the query 3 times concurrently: Execution # Elapsed Time/s Killed Task Attempts
----------- -------------- --------------------
1 90 1
2 159 110
3 190 182
I'd be very grateful for guidance on what is happening and why.
Am I hitting https://issues.apache.org/jira/browse/HIVE-15779?
Thanks,
Martin
... View more
- Tags:
- concurrency
- llap
07-01-2015
01:25 AM
Thanks for the response. I have been meaning to come back and add a comment here. I found the root of the problem as well as the solution here: http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/kerberos-authentication-failure-GSSAPI-Failure-gss-accept-sec/td-p/23333
... View more