Thank you, @Pardeep and @Sagar Shimpi! Finally, from the articles above and from your replies I have managed to get a short summary. I am posting this in case somebody else wonders about this 🙂 "In order to handle the variety of
workloads related with intense CPU usage, YARN has introduced a new concept
called "vcores" (short for virtual cores). A vcore, is a usage share
of a host CPU which YARN Node Manager allocates to use all available resources
in the most efficient possible way. YARN hosts can be tuned to optimize the use
of vcores by configuring the available YARN containers as the number of vcores
has to be set by an administrator in yarn-site.xml on each node. The decision
of how much it should be set to is driven by the type of workloads running in
the cluster and the type of hardware available. The general recommendation is
to set it to the number of physical cores on the node, but administrators can
bump it up if they wish to run additional containers on nodes with faster CPUs.
In order to enable CPU scheduling, there are some configuration properties that
administrators and users need to be aware of:
yarn.nodemanager.resource.cpu-vcores:
Set to the appropriate number in yarn-site.xml on all the nodes. This is
strictly dependent on the type of workloads running in a cluster, but the
general recommendation is that admins set it to be equal to the number of
physical cores on the machine. yarn.scheduler.minimum-allocation-vcores:
This is the minimum allocation for every container request at the Resource
Manager, in terms of virtual CPU cores. Requests lower than this won't take
effect, and the specified value will get allocated the minimum. yarn.scheduler.maximum-allocation-vcores:
This is the maximum allocation for every container request at the Resource
Manager, in terms of virtual CPU cores. Requests higher than this won't take
effect, and will get capped to this value. “yarn.scheduler.maximum-allocation-vcores”
controls the maximum vcores that any submitted job can request.
“yarn.nodemanager.resource.cpu-vcores” controls how many vcores can be
scheduled on a particular NodeManager instance. So “yarn.nodemanager.resource.cpu-vcores”
can vary from host to host (NodeManager to NodeManager), while
“yarn.scheduler.maximum-allocation-vcores” is a global property of the
scheduler."
Further information can be taken also from here: https://community.cloudera.com/t5/Cloudera-Manager-Installation/yarn-nodemanager-resource-cpu-vcores-and-yarn-scheduler-maximum/td-p/31098
... View more
