Created on 08-23-2015 10:11 AM - edited 09-16-2022 02:38 AM
These two default parametes doesnt make sense for me.
yarn.nodemanager.resource.cpu-vcores:
the total available cores for all containers.
"Number of CPU cores that can be allocated for containers."
yarn.scheduler.maximum-allocation-vcores:
the maximum number of cores that a container can request.
"The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value."
So if I have only 8 cores available for all containers how can the AM request a 32 core container?
These two default parameters must bem at least the same, or the yarn.scheduler.maximum-allocation-vcores less than yarn.nodemanager.resource.cpu-vcores.
Am i rigth?
Created 08-24-2015 03:53 PM
Hi -
yarn.scheduler.maximum-allocation-vcores controls the maximum vcores that any submitted job can request. yarn.nodemanager.resource.cpu-vcores, on the other hand, controls how many vcores can be scheduled on a particular NodeManager instance.
So yarn.nodemanager.resource.cpu-vcores can vary from host to host (NodeManager to NodeManager), while yarn.scheduler.maximum-allocation-vcores is a global property of the scheduler.
Regards,
Mark
Yes... it doesn't make sense... I bumped into the same question, and the default of "yarn.scheduler.maximum-allocation-vcores" is actually 4 in the code.
I fired a JIRA for this.
https://issues.apache.org/jira/browse/YARN-4499