LLAP capacity

[#1]

Hello All 

i would like to understand what does it mean that - once i start LLAP i can see below information in YARN resource manager:

LLAP is running, no jobs/queries executed but process itself takes 81 % of queue (not cluster), what is consuming this 81% ? 

application_1569522555597_2379

hive

llap0

org-apache-slider

llap

1

Fri Oct 11 13:40:54 +0200 2019

N/A

RUNNING

UNDEFINED

6

6

241664

81.9

51.8

0

cheers 

[#2]

@stryjz - LLAP is Live Long and Process. So the LLAP daemon will keep on running irrespective of whether you are running query or not. It consists of a long-lived daemon which replaces direct interactions with the HDFS Data Node, and a tightly integrated DAG-based framework.
So based on the LLAP Daemon size that particular YARN application will be utilizing the resources. You can refer below articles for more information on this:
https://cwiki.apache.org/confluence/display/Hive/LLAP
https://community.cloudera.com/t5/Community-Articles/Hive-LLAP-deep-dive/ta-p/248893

[#3]

thanks for your reply @ngarg 

I am not sure if you got my point. 

LLAP is showing that it is consuming 99% of the resources in the queue (own queue) 

And if this is the case there is not enough memory for AM - at least this is my assumption.

So my question is - if this is desired to see that LLAP app is showing so high usage of its own queue just by running LLAP app/daemon?  with below(table "not working LLAP") all queries are in the accepted state waiting for resources.

Once we changed Memory per Daemon to use more/less 3gb memory less than suggested by Ambari, LLAP started to give us results 

Both tables below are showing LLAP while no running queries 

Not working LLAP 

Used Capacity:Configured Capacity:Configured Max Capacity:Absolute Used Capacity:Absolute Configured Capacity:Absolute Configured Max Capacity:Used Resources:Configured Max Application Master Limit:Max Application Master Resources:Used Application Master Resources:Max Application Master Resources Per User:

99.3%
100.0%
100.0%
99.3%
100.0%
100.0%
<memory:232448, vCores:10>
100.0
<memory:294912, vCores:36>
<memory:17408, vCores:5>
<memory:294912, vCores:36>

Working LLAP 

Used Capacity:Configured Capacity:Configured Max Capacity:Absolute Used Capacity:Absolute Configured Capacity:Absolute Configured Max Capacity:Used Resources:Configured Max Application Master Limit:Max Application Master Resources:Used Application Master Resources:Max Application Master Resources Per User:

78.8%
100.0%
100.0%
78.8%
100.0%
100.0%
<memory:232448, vCores:10>
100.0
<memory:294912, vCores:36>
<memory:17408, vCores:5>
<memory:294912, vCores:36>

[#4]

@stryjz - In case of "Not working LLAP" queue utilization is almost 100%. Due to which you are not able to run any queries. In this case you can either increase the queue size or tune your LLAP settings. You can check below article for tuning LLAP.

https://community.cloudera.com/t5/Community-Articles/LLAP-sizing-and-setup/ta-p/247425

[#5]

The query execution was not working in the first case i.e. when the used capacity was close 100%, as there was no capacity left in the queue to start the Query Coordinator App/Apps.

And Query Coordinator App is a must to run the query on LLAP.

Without a Query Coordinator app, you will be able to connect to LLAP, but no query will run(which requires a Tez task).

There is a property in Hive "Maximum Total Concurrent Queries", this determines how many more apps would be running under the LLAP queue.

These apps are for co-ordinating between the different mapper and reducer tasks running under LLAP Daemons(in llap0 app).

Thus, if you have LLAP configured with below details, it will not work for a 100GB "llap" Yarn Queue capacity, it will require a minimum of 108GB "llap" Yarn Queue capacity(assuming that default AM size is 4 GB):

Memory per Daemon=50GB

Number of daemons=50GB

Maximum Total Concurrent Queries=1

In the above case, the "llap0" application will need approx. 104GB(100GB for 2 daemons and some extra amount for its AM usually 4GB). The "llap0" always runs with (number of Daemons+1) containers. The extra 1 for AM. In this case as the Number of Daemons are 2, the "llap0" app will run with 3 containers(2 containers of 50GB each and one AM container).

Apart from the above "llap0", there will be a Tez application with the name such as "HIVE-b52bb8c2-db60-4a79-9526-e61c9ee48261" (example) with in the "llap" yarn queue.

This is the query co-ordinator app, as "Maximum Total Concurrent Queries" is set to 1, you will only see one such application. if the "Maximum Total Concurrent Queries" is set to 2, you will see 2 such apps and so on.

Thus, you "llap" Yarn queue will need 4GB(default AM size) more for Query coordinator app to start.

So, the total requirement for "llap" Yarn queue will be 104GB+4GB=108GB.

Total Yarn queue requirement for LLAP= {(Number of Daemons)*(Memory per Daemon)+Default AM Size} + {Maximum Total Concurrent Queries*Default AM Size}

The first part is "llap0" app and second part is the number of query coordinator apps in the above formula.

Hope I have explained it well!!

Let me know if you get confused!