Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores

yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores

New Contributor

These two default parametes doesnt make sense for me.
yarn.nodemanager.resource.cpu-vcores:
the total available cores for all containers.
"Number of CPU cores that can be allocated for containers."

yarn.scheduler.maximum-allocation-vcores:
the maximum number of cores that a container can request.
"The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value."

So if I have only 8 cores available for all containers how can the AM request a 32 core container?

These two default parameters must bem at least the same, or the yarn.scheduler.maximum-allocation-vcores less than yarn.nodemanager.resource.cpu-vcores.

Am i rigth?

2 REPLIES 2

Re: yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores

Cloudera Employee

Hi -

 

yarn.scheduler.maximum-allocation-vcores controls the maximum vcores that any submitted job can request. yarn.nodemanager.resource.cpu-vcores, on the other hand, controls how many vcores can be scheduled on a particular NodeManager instance. 

 

So yarn.nodemanager.resource.cpu-vcores can vary from host to host (NodeManager to NodeManager), while yarn.scheduler.maximum-allocation-vcores is a global property of the scheduler.

 

Regards,

Mark

 

Re: yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores

New Contributor

Yes... it doesn't make sense... I bumped into the same question, and the default of "yarn.scheduler.maximum-allocation-vcores" is actually 4 in the code. 

I fired a JIRA for this.
https://issues.apache.org/jira/browse/YARN-4499