Reply
New Contributor
Posts: 1
Registered: ‎08-23-2015

yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores

These two default parametes doesnt make sense for me.
yarn.nodemanager.resource.cpu-vcores:
the total available cores for all containers.
"Number of CPU cores that can be allocated for containers."

yarn.scheduler.maximum-allocation-vcores:
the maximum number of cores that a container can request.
"The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value."

So if I have only 8 cores available for all containers how can the AM request a 32 core container?

These two default parameters must bem at least the same, or the yarn.scheduler.maximum-allocation-vcores less than yarn.nodemanager.resource.cpu-vcores.

Am i rigth?

Cloudera Employee
Posts: 8
Registered: ‎05-15-2015

Re: yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores

Hi -

 

yarn.scheduler.maximum-allocation-vcores controls the maximum vcores that any submitted job can request. yarn.nodemanager.resource.cpu-vcores, on the other hand, controls how many vcores can be scheduled on a particular NodeManager instance. 

 

So yarn.nodemanager.resource.cpu-vcores can vary from host to host (NodeManager to NodeManager), while yarn.scheduler.maximum-allocation-vcores is a global property of the scheduler.

 

Regards,

Mark

 

Highlighted
New Contributor
Posts: 1
Registered: ‎12-22-2015

Re: yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores

[ Edited ]

Yes... it doesn't make sense... I bumped into the same question, and the default of "yarn.scheduler.maximum-allocation-vcores" is actually 4 in the code. 

I fired a JIRA for this.
https://issues.apache.org/jira/browse/YARN-4499

Announcements