Support Questions

Find answers, ask questions, and share your expertise

Limit vcores with Capacity Scheduler

avatar

I am running a cluster with Capacity Scheduler. Can I limit vcores for a queue with concrete number of vcores, not percentages? I found option yarn.scheduler.capacity.<queue_path>.maximum-allocation-vcores. But it seems like it doesn't work. I added this option into "Capacity Scheduler Configuration Advanced Configuration Snippet (Safety Valve)" in Cloudera Manager, but applications still use more than this restriction. Can someone explain why?

1 REPLY 1

avatar

After careful exploration of logs I have found some strange behavior and cant explain it. After restart there are maximum 2 vcores, that can be allocated for this queue. However, when I launch jobs, I see more vcores allocated than max for this queue. Screenshots below are made one after another. Why does it happen?Screen Shot 2020-10-15 at 1.17.40 PM.pngScreen Shot 2020-10-15 at 1.16.48 PM.png