Created on 10-30-2023 01:14 PM - edited 10-30-2023 01:16 PM
Hi everyone.
I created 2 queue yarns in cdp 7.1.8.
Precisely I created 2 queues:
users queue --> configured capacity 50% and maximum capacity 100%
hive queue -->configured capacity 50% and maximum capacity 100%
I enabled child queues mode for all 2 queues with the following configuration parameters:
Dynamic Queue Minimum User Limit | 100% |
Dynamic Queue User Limit Factor | 1 |
Dynamic Queue Maximum Applications | 1000 |
Dynamic Queue Maximum AM Resource Limit | 20% |
Dynamic Queue Ordering Policy | Fair |
The problem we encounter is that if even just 2 queries are executed in the same queue the second goes pending and is not executed in parallel, I show an example below
Another example would be an oozie job that only calls a spark job. The oozie job runs infinitely waiting for the pyspark to run. However, in the default queue this problem is not present.The development cluster is made up of 3 worker nodes (144 vcores 180 gb yarn memory).
Can you provide us support or optimizations to do at the queue configuration level?
Thanks in advance
Lorenzo
Created 11-20-2023 01:12 AM
I temporarily solved it by eliminating the dynamic child creation
Created 11-02-2023 03:00 AM
Try increasing the Dynamic Queue User Limit Factor to 2 or 4 and check if that helps.
Created 11-20-2023 01:12 AM
I temporarily solved it by eliminating the dynamic child creation
Created 11-06-2023 12:44 PM
@Lorenzo Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
Regards,
Diana Torres,Created 05-08-2024 05:52 AM
I temporarily solved it by eliminating the dynamic child creation