yarn.nodemanager.resource.cpu-vcores is by default coming up with value multiplied by 2

[#1]

We have cluster with below CPU configuration:

# lscpu

Architecture: x86_64

CPU op-mode(s): 32-bit, 64-bit

Byte Order: Little Endian

CPU(s): 56

On-line CPU(s) list: 0-55

Thread(s) per core: 2

Core(s) per socket: 14

Socket(s): 2

NUMA node(s): 2

Vendor ID: GenuineIntel

CPU family: 6

Model: 79

Stepping: 1

CPU MHz: 2400.000

BogoMIPS: 4794.00

Virtualization: VT-x

L1d cache: 32K

L1i cache: 32K

L2 cache: 256K

L3 cache: 35840K

NUMA node0 CPU(s): 0-13,28-41

NUMA node1 CPU(s): 14-27,42-55

We have 2 Physical Cores, 14 CPU's each, with Hyper threading 2 (physical) * 14 (cpu's each) * 2 (hyder threading) = 56

But the YARN Configs from Ambari shows 112 cores for the property yarn.nodemanager.resource.cpu-vcores (56*2)

This is being done at the Ambari Stack Advisor Code

The question here is are we by default doing the multiplication by 2 by assuming that the Hyper Threading is not enabled or are we considering that the CPU is capable of holding multiple containers and leaving the scope to admins to tune the environment based on CPU or I/O work loads.

Thanks

Venkat

[#2 · Accepted solution]

Hi @Venkata Sudheer Kumar M CPU is capable of running multiple containers if the jobs are not cpu intensive.

The stack advisor is only recommending to not go beyond "CPU(s) * 2". However, there is nothing stopping you from configuring higher.

if you observe your container concurrency metrics and CPU utilization, you can identify your threshold of vcores to 1 CPU and set it accordingly. Note: yarn.nodemanager.resource.cpu-vcores would only be applicable if you enable CPU Scheduling.

[#3]

@Saumil Mayani Thanks a lot for the details. That makes it more clear.