Hive llap performance issue.

[#1]

We have HDP 2.6.2.14 version installed. Enable the LLAP and it was working very fine till 4 month. Now we are facing issue for performance. Earlier same query was taking around 51 sec and now it is taking more than 9 min's.

We haven't change anything on server and configuration.

Immediate help will be appreciated.

Thanks in advance.

[#2]

@Ashnee Sharma

Is there a change in the data volume? Also, what is the performance for the consecutive query runs?

[#3]

@Sindhu

No change in volume and we were running continuous query on llap like 3-7 queries in parallel. Same thing we were doing and now also but it is taking time.

My obserbation is it is taking time for reducer's.

[#4]

@Sindhu

And consecutive query performance also decreased. Please suggest it I can tune the reducer's.

I have tried to tune my cluster with following parameter.

MemPerDaemon (Container Size) > LLAP Heapsize (Java process heap) + CacheSize (off heap) +headroom

Multiple of yarn min allocation Should be less than yarn.nodemanager.resource.memory-mb Headroom is capped at 6GB QueueSize (Yarn Queue) >= MemPerDaemon * num daemons + slider + (tez AM size * concurrency)

Cachesize = MemPerDaemon - (hive tez container * num of executors)

Num executors per daemon = (MemPerDaemon - cache_size)/hive tez container size

Our cluster size is 4 datanode and 1 master.