Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Inability to execute multiple queries from the same user after configuring yarn queue

avatar
Contributor

Hi everyone.

I created 2 queue yarns in cdp 7.1.8.

Precisely I created 2 queues:

users queue --> configured capacity 50% and maximum capacity 100%

hive queue -->configured capacity 50% and maximum capacity 100%

I enabled child queues mode for all 2 queues with the following configuration parameters:

Dynamic Queue Minimum User Limit100%
Dynamic Queue User Limit Factor1
Dynamic Queue Maximum Applications1000
Dynamic Queue Maximum AM Resource Limit20%
Dynamic Queue Ordering PolicyFair

The problem we encounter is that if even just 2 queries are executed in the same queue the second goes pending and is not executed in parallel, I show an example belowtest.PNG

Another example would be an oozie job that only calls a spark job. The oozie job runs infinitely waiting for the pyspark to run. However, in the default queue this problem is not present.The development cluster is made up of 3 worker nodes (144 vcores 180 gb yarn memory).

Can you provide us support or optimizations to do at the queue configuration level?

Thanks in advance

Lorenzo

1 ACCEPTED SOLUTION

avatar
Contributor

I temporarily solved it by eliminating the dynamic child creation

View solution in original post

4 REPLIES 4

avatar
Rising Star

Try increasing the Dynamic Queue User Limit Factor to 2 or 4 and check if that helps.

avatar
Contributor

I temporarily solved it by eliminating the dynamic child creation

avatar
Community Manager

@Lorenzo Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Contributor
 
I temporarily solved it by eliminating the dynamic child creation