how to calculate the yarn.nodemanager.resource.cpu-vcores value when Thread(s) per core is 2

[#1]

hi all,

Number of vcores ( yarn.nodemanager.resource.cpu-vcores , should be around ~80% of total vCPUs available on datanode machine machine,

but what in case we have on datanode machine the following info :

32 core ,

and Thread(s) per core is 2

lscpu

Thread(s) per core:    2

in this case what should be the yarn.nodemanager.resource.cpu-vcores ?

is it 80% from 64 ?

or

80% from 32 ?

[#2]

@Michael Bronson

Adding @Tarun Parimi . Tarun, Can you please share your thoughts on this.

[#3]

@Michael Bronson

The lscpu output of "CPU(s):" itself always takes into consideration the "Thread(s) per core".

So usually, CPU(s) = [Thread(s) per core] X [Core(s) per socket] X [Socket(s)]

It is sufficient to consider only CPU(s), when setting yarn.nodemanager.resource.cpu-vcores .

[#4]

@Tarun , first thank you , but I will give you interesting example

we have ambari cluster with 3 datanode

each data node have 32 Vcore with 2 thread ( as we get from lscpu )

so according to your answer we need to set 32 to yarn.nodemanager.resource.cpu-vcores

but as you can see from the following picture

the scrollbar is until 64Vcore , and seems that during ambari installation , ambari calculate 2 X 32 and that gives us the scroll-bar to 64

so regarding my example , do you still with opinion that we need to set 80% of 32 ?

[#5]

@Michael Bronson Yarn Vcores can ideally be set up to 2x the actual cpu present based on the use case. Thats why ambari provides the option in the scroll bar. It does not depend on the number of threads shown in lscpu. If you want to prevent over utilization of cpu by Yarn and leave cpu for OS and other processes you can set to 80% of 32 . But keep in mind that this value will only be considered by YARN for scheduling containers, when CPU scheduling is enabled.

[#6]

@Tarun , so the final answer is to set 80% from 32

second , I will give you interesting info

some of our ambari clusters the scrollbar is limit as the vcore ( for example if we have 16 vcore then scroll bar is until 16

and some cluster the scrolbar is twice then the real vcore ( for example if we have 16 vcore the scroll bar is 32 )

so

how you explain this?

[#7]

Interesting. Can you paste the lscpu output of the nodes you are mentioning?

[#8]

@Tarun , sorry , yes you rught all cluster are twice then the Vcore value , its my fault

[#9]

@Tarun , can we ask you to post article about this ? , to explain what we should set as you said here , I think this will be useful to explain all aspects because some users think that the number of thread in lscpu needed to consider to calculate the Vcore

[#10]

@Michael Bronson I will look to create an article about configuring the vcores for cpu scheduling when I get time. I will mention this part there.