- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
how to calculate the yarn.nodemanager.resource.cpu-vcores value when Thread(s) per core is 2
Created 09-12-2018 03:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
hi all,
Number of vcores ( yarn.nodemanager.resource.cpu-vcores , should be around ~80% of total vCPUs available on datanode machine machine,
but what in case we have on datanode machine the following info :
32 core ,
and Thread(s) per core is 2
lscpu Thread(s) per core: 2
in this case what should be the yarn.nodemanager.resource.cpu-vcores ?
is it 80% from 64 ?
or
80% from 32 ?
Created 09-13-2018 08:39 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created 09-13-2018 09:28 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The lscpu output of "CPU(s):" itself always takes into consideration the "Thread(s) per core".
So usually, CPU(s) = [Thread(s) per core] X [Core(s) per socket] X [Socket(s)]
It is sufficient to consider only CPU(s), when setting yarn.nodemanager.resource.cpu-vcores .
Created on 09-13-2018 10:00 AM - edited 08-18-2019 12:48 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Tarun , first thank you , but I will give you interesting example
we have ambari cluster with 3 datanode
each data node have 32 Vcore with 2 thread ( as we get from lscpu )
so according to your answer we need to set 32 to yarn.nodemanager.resource.cpu-vcores
but as you can see from the following picture
the scrollbar is until 64Vcore , and seems that during ambari installation , ambari calculate 2 X 32 and that gives us the scroll-bar to 64
so regarding my example , do you still with opinion that we need to set 80% of 32 ?
Created 09-13-2018 10:12 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Michael Bronson Yarn Vcores can ideally be set up to 2x the actual cpu present based on the use case. Thats why ambari provides the option in the scroll bar. It does not depend on the number of threads shown in lscpu. If you want to prevent over utilization of cpu by Yarn and leave cpu for OS and other processes you can set to 80% of 32 . But keep in mind that this value will only be considered by YARN for scheduling containers, when CPU scheduling is enabled.
Created 09-13-2018 10:31 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Tarun , so the final answer is to set 80% from 32
second , I will give you interesting info
some of our ambari clusters the scrollbar is limit as the vcore ( for example if we have 16 vcore then scroll bar is until 16
and some cluster the scrolbar is twice then the real vcore ( for example if we have 16 vcore the scroll bar is 32 )
so
how you explain this?
Created 09-13-2018 10:48 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Interesting. Can you paste the lscpu output of the nodes you are mentioning?
Created 09-13-2018 11:06 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Tarun , sorry , yes you rught all cluster are twice then the Vcore value , its my fault
Created 09-13-2018 11:25 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Tarun , can we ask you to post article about this ? , to explain what we should set as you said here , I think this will be useful to explain all aspects because some users think that the number of thread in lscpu needed to consider to calculate the Vcore
Created 09-13-2018 12:02 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Michael Bronson I will look to create an article about configuring the vcores for cpu scheduling when I get time. I will mention this part there.