Hi,
I'm writing to you because I have some trouble or misunderstanding with Hive LLAP.
LLAP is slow when executing queries and I feel like LLAP is not well configured. I checked the jmx metrics of my LLAP daemons and I saw that :
"ExecutorTotalRequestsHandled" : 89065,
"ExecutorTotalSuccess" : 22893,
"ExecutorTotalFailed" : 58,
"ExecutorTotalKilled" : 21724,
"ExecutorTotalEvictedFromWaitQueue" : 43678,
"ExecutorTotalRejectedRequests" : 406234,
"ExecutorTotalPreemptionTimeToKill" : 73450159,
"ExecutorTotalPreemptionTimeLost" : 1083694998,
Each deamon has the same behavior.
Most of executed tasks are killed or evicted (only 25% of sucess tasks...) and I think this is one of the reason of the slowness.
Below my configuration :
- 75 nodes reserved for LLAP (node labelling).
- 280 GB - 40 CPUs per node.
- 220 GB - 36 CPU per daemon.
- Daemon : 144 GB for the heap, 60 GB for the cache, 16 GB for the headroom.
- hive.tez.container.size = 4 G
- tez.am.resource.memory.mb = 12 G
- hive.llap.io.threadpool.size = 36
Does this configuration look well for you ?
Also, when I check the execution of some queries, I can see that :
Map 1: 855(+40426,-1167)/41281 Reducer 2: 0/1009
It means that "Map 1" has 41281 tasks map, 855 are completed and 40426 are running (and 1167 are pending) but how it that possible because I have 75 daemon and 36 CPUs per deamon (75*36 = 2700), So I should have at most 2700 tasks running (1 CPU per task), no ?
What do you think ? Any ideas, suggestions or questions would be useful.
Thanks a lot for your help.
