question Inability to execute multiple queries from the same user after configuring yarn queue in Support Questions

Inability to execute multiple queries from the same user after configuring yarn queue

Lorenzo — Mon, 30 Oct 2023 20:16:51 GMT

Hi everyone.

I created 2 queue yarns in cdp 7.1.8.

Precisely I created 2 queues:

users queue --> configured capacity 50% and maximum capacity 100%

hive queue -->configured capacity 50% and maximum capacity 100%

I enabled child queues mode for all 2 queues with the following configuration parameters:

Dynamic Queue Minimum User Limit	100%
Dynamic Queue User Limit Factor	1
Dynamic Queue Maximum Applications	1000
Dynamic Queue Maximum AM Resource Limit	20%
Dynamic Queue Ordering Policy	Fair

The problem we encounter is that if even just 2 queries are executed in the same queue the second goes pending and is not executed in parallel, I show an example below

Another example would be an oozie job that only calls a spark job. The oozie job runs infinitely waiting for the pyspark to run. However, in the default queue this problem is not present.The development cluster is made up of 3 worker nodes (144 vcores 180 gb yarn memory).

Can you provide us support or optimizations to do at the queue configuration level?

Thanks in advance

Lorenzo

Re: Inability to execute multiple queries from the same user after configuring yarn queue

Asok — Thu, 02 Nov 2023 10:00:21 GMT

Try increasing the Dynamic Queue User Limit Factor to 2 or 4 and check if that helps.

Re: Inability to execute multiple queries from the same user after configuring yarn queue

DianaTorres — Mon, 06 Nov 2023 20:44:46 GMT

@Lorenzo Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

Re: Inability to execute multiple queries from the same user after configuring yarn queue

Lorenzo — Mon, 20 Nov 2023 09:12:28 GMT

I temporarily solved it by eliminating the dynamic child creation

Re: Inability to execute multiple queries from the same user after configuring yarn queue

Lorenzo — Wed, 08 May 2024 12:52:15 GMT

I temporarily solved it by eliminating the dynamic child creation