Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores

avatar
New Contributor

These two default parametes doesnt make sense for me.
yarn.nodemanager.resource.cpu-vcores:
the total available cores for all containers.
"Number of CPU cores that can be allocated for containers."

yarn.scheduler.maximum-allocation-vcores:
the maximum number of cores that a container can request.
"The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value."

So if I have only 8 cores available for all containers how can the AM request a 32 core container?

These two default parameters must bem at least the same, or the yarn.scheduler.maximum-allocation-vcores less than yarn.nodemanager.resource.cpu-vcores.

Am i rigth?

2 REPLIES 2

avatar
Cloudera Employee

Hi -

 

yarn.scheduler.maximum-allocation-vcores controls the maximum vcores that any submitted job can request. yarn.nodemanager.resource.cpu-vcores, on the other hand, controls how many vcores can be scheduled on a particular NodeManager instance. 

 

So yarn.nodemanager.resource.cpu-vcores can vary from host to host (NodeManager to NodeManager), while yarn.scheduler.maximum-allocation-vcores is a global property of the scheduler.

 

Regards,

Mark

 

avatar
New Contributor

Yes... it doesn't make sense... I bumped into the same question, and the default of "yarn.scheduler.maximum-allocation-vcores" is actually 4 in the code. 

I fired a JIRA for this.
https://issues.apache.org/jira/browse/YARN-4499