Calculating Minimum Queue Capacity required for starting Hive's LLAP

[#1]

If the cluster has only one queue at root level named 'default' and is consuming 100% of the capacity, Ambari will create a queue named 'llap' when HSI is enabled for the 1st time, which is set to (depends on which value is smaller) either :

- the minimum required %age for LLAP to work, or

- at 20% of cluster's capacity.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

If this is not the case, where there are more than one queue in cluster, user will have to create/set the queue capacity %age in order to be used for LLAP app.

Starting with minimum required for queue capacity (shown below), one can increase the queue %age size in order to add up the LLAP nodes in the cluster, as queue size is one of the primary drivers of how many Node Managers nodes will be running LLAP.

Reference code for calculating minimum queue size.

Following calculations can be a good reference in order to calculate the minimum queue capacity %age to be set by using the following config values as referenced from Ambari UI :

- Total Node Manager nodes in Ambari cluster (NMCount). Can be got from Ambari's YARN page.

- YARN Node Manager Size (YarnNMSize) (yarn-site/yarn.nodemanager.resource.memory-mb)

- YARN minimum container size (YarnMinContSize) (yarn-site/yarn.scheduler.minimum-allocation-mb)

- Slider AM container size (SliderAmSize) (hive-interactive-env/slider_am_container_mb). It is calculated as shown here.

- Hive Tez Container Size (HiveTezContSize) (hive-interactive-site/hive.tez.container.size)

- Tez AM container size (TezAmContSize) (tez-interactive-site/tez.am.resource.memory.mb)

NormalizeUp() function is used to normalize the 1st parameter w.r.t. 2nd parameter (YarnMinContSize). Code reference is here, where the snippet function can be used for calculating by putting in a python file and called with correct params, or doing a manual calculation.

Min. Total capacity required for queue to run LLAP (MinCapForLlapQueue) = 
                                                NormalizeUp(SliderAmSize, YarnMinContSize) + 
                                                NormalizeUp(HiveTezContSize, YarnMinContSize) + 
                                                NormalizeUp(TezAmContSize, YarnMinContSize)

Total Cluster Capacity (ClusterCap) = NMCount * YarnNMSize

Min. Queue Percentage Required for queue used for LLAP (in %) (MinQueuePerc) = MinCapForLlapQueue * 100 / ClusterCap

Thus, 'MinQueuePerc' value can used to set the queue size to be used for LLAP app.

The queue %age can be changed from Ambari > Views > YARN Queue Manager.