Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Yarn Physical to Virtual Core multiplier

avatar
Explorer

We use CDH version 5.7.1 and Cloudera Manager 5.8.

 

In Yarn tuning Excel, I see a field called "Physical Cores to Vcores Multiplier" which has the following description:

 

"Set this ratio based on the expected number of concurrent threads per core. Use 1 for CPU intensive tasks up to 4 for standard I/O bound tasks."

 

I could not find a Cloudera Manager setting corresponding to it. How do we tell Yarn that it can use a multiplier of 4 ? Is there a default value to it ?

 

Or is this multiplier indicated to Yarn by setting "yarn.nodemanager.resource.cpu-vcores" parameter to a desired value instead of just the count of physical core ?

 

Say, if I have 24 physical cores, and I intend to use 50% of it for Yarn, then should I put 12 for this parameter to indicate that I want a ratio of 1:1 and put 48 to indicate a ratio of 1:4 ?

 

We use static pool resource allocation for various Hadoop components (Hbase, Solr, Hadoop, Yarn) and we set ~50% of resources to Yarn. But I could not find the multiplier anywhere.

 

Thanks,

Vivek

1 ACCEPTED SOLUTION

avatar
Super Collaborator

The "multiplier" is not a parameter.

You directly set the number of vcpu you want yarn to use. I guess you have to do the math yourself before setting the value.

View solution in original post

4 REPLIES 4

avatar
Super Collaborator

The "multiplier" is not a parameter.

You directly set the number of vcpu you want yarn to use. I guess you have to do the math yourself before setting the value.

avatar
Explorer

Thank you Mathieu.d

 

Is this the vcpu parameter you are referring to ? yarn.nodemanager.resource.cpu-vcores

 

Thanks,

Vivek

avatar
Super Collaborator

Yes, this one.

avatar
Explorer
Thanks for confirming. I appreciate your quick feedback.