Inability to execute multiple queries from the same user after configuring yarn queue

[#1]

Hi everyone.

I created 2 queue yarns in cdp 7.1.8.

Precisely I created 2 queues:

users queue --> configured capacity 50% and maximum capacity 100%

hive queue -->configured capacity 50% and maximum capacity 100%

I enabled child queues mode for all 2 queues with the following configuration parameters:

Dynamic Queue Minimum User Limit	100%
Dynamic Queue User Limit Factor	1
Dynamic Queue Maximum Applications	1000
Dynamic Queue Maximum AM Resource Limit	20%
Dynamic Queue Ordering Policy	Fair

The problem we encounter is that if even just 2 queries are executed in the same queue the second goes pending and is not executed in parallel, I show an example below

Another example would be an oozie job that only calls a spark job. The oozie job runs infinitely waiting for the pyspark to run. However, in the default queue this problem is not present.The development cluster is made up of 3 worker nodes (144 vcores 180 gb yarn memory).

Can you provide us support or optimizations to do at the queue configuration level?

Thanks in advance

Lorenzo

[#2]

Try increasing the Dynamic Queue User Limit Factor to 2 or 4 and check if that helps.

[#3 · Accepted solution]

I temporarily solved it by eliminating the dynamic child creation

[#4]

@Lorenzo Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

[#5]

I temporarily solved it by eliminating the dynamic child creation