Support Questions
Find answers, ask questions, and share your expertise

Hive LLAP - configuration - slowness - tasks killed

Explorer

Hi,

 

I'm writing to you because I have some trouble or misunderstanding with Hive LLAP.

LLAP is slow when executing queries and I feel like LLAP is not well configured. I checked the jmx metrics of my LLAP daemons and I saw that :

 

"ExecutorTotalRequestsHandled" : 89065,
"ExecutorTotalSuccess" : 22893,
"ExecutorTotalFailed" : 58,
"ExecutorTotalKilled" : 21724,
"ExecutorTotalEvictedFromWaitQueue" : 43678,
"ExecutorTotalRejectedRequests" : 406234,
"ExecutorTotalPreemptionTimeToKill" : 73450159,
"ExecutorTotalPreemptionTimeLost" : 1083694998,

 

Each deamon has the same behavior.

 

Most of executed tasks are killed or evicted (only 25% of sucess tasks...) and I think this is one of the reason of the slowness.

 

Below my configuration :

- 75 nodes reserved for LLAP (node labelling).

- 280 GB - 40 CPUs per node.

- 220 GB - 36 CPU per daemon.

- Daemon : 144 GB for the heap, 60 GB for the cache, 16 GB for the headroom.

- hive.tez.container.size = 4 G

- tez.am.resource.memory.mb = 12 G

- hive.llap.io.threadpool.size = 36

 

Does this configuration look well for you ?

 

Also, when I check the execution of some queries, I can see that :

 

Map 1: 855(+40426,-1167)/41281 Reducer 2: 0/1009

 

It means that "Map 1" has 41281 tasks map, 855 are completed and 40426 are running (and 1167 are pending) but how it that possible because I have 75 daemon and 36 CPUs per deamon (75*36 = 2700), So I should have at most 2700 tasks running (1 CPU per task), no ?

 

What do you think ? Any ideas, suggestions or questions would be useful.

 

Thanks a lot for your help.

 

2 REPLIES 2

Expert Contributor

@Simon230 What is the value for Memory Per Daemon and how many executors do you have ?

Explorer

@asish 

- 220 GB - 36 executors per daemon.

- in a daemon : 144 GB for the heap, 60 GB for the cache, 16 GB for the headroom.

; ;