Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

yarn.nodemanager.resource.cpu-vcores is by default coming up with value multiplied by 2

Solved Go to solution
Highlighted

yarn.nodemanager.resource.cpu-vcores is by default coming up with value multiplied by 2

Expert Contributor

We have cluster with below CPU configuration:

# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-55
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Stepping: 1
CPU MHz: 2400.000
BogoMIPS: 4794.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 35840K
NUMA node0 CPU(s): 0-13,28-41
NUMA node1 CPU(s): 14-27,42-55

We have 2 Physical Cores, 14 CPU's each, with Hyper threading 2 (physical) * 14 (cpu's each) * 2 (hyder threading) = 56

But the YARN Configs from Ambari shows 112 cores for the property yarn.nodemanager.resource.cpu-vcores (56*2)

This is being done at the Ambari Stack Advisor Code

The question here is are we by default doing the multiplication by 2 by assuming that the Hyper Threading is not enabled or are we considering that the CPU is capable of holding multiple containers and leaving the scope to admins to tune the environment based on CPU or I/O work loads.

Thanks

Venkat

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: yarn.nodemanager.resource.cpu-vcores is by default coming up with value multiplied by 2

Expert Contributor

Hi @Venkata Sudheer Kumar M CPU is capable of running multiple containers if the jobs are not cpu intensive.

The stack advisor is only recommending to not go beyond "CPU(s) * 2". However, there is nothing stopping you from configuring higher.

if you observe your container concurrency metrics and CPU utilization, you can identify your threshold of vcores to 1 CPU and set it accordingly. Note: yarn.nodemanager.resource.cpu-vcores would only be applicable if you enable CPU Scheduling.

View solution in original post

2 REPLIES 2
Highlighted

Re: yarn.nodemanager.resource.cpu-vcores is by default coming up with value multiplied by 2

Expert Contributor

Hi @Venkata Sudheer Kumar M CPU is capable of running multiple containers if the jobs are not cpu intensive.

The stack advisor is only recommending to not go beyond "CPU(s) * 2". However, there is nothing stopping you from configuring higher.

if you observe your container concurrency metrics and CPU utilization, you can identify your threshold of vcores to 1 CPU and set it accordingly. Note: yarn.nodemanager.resource.cpu-vcores would only be applicable if you enable CPU Scheduling.

View solution in original post

Highlighted

Re: yarn.nodemanager.resource.cpu-vcores is by default coming up with value multiplied by 2

Expert Contributor

@Saumil Mayani Thanks a lot for the details. That makes it more clear.

Don't have an account?
Coming from Hortonworks? Activate your account here