Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Physical to Virtual Core multiplier

Solved Go to solution

Yarn Physical to Virtual Core multiplier

Explorer

We use CDH version 5.7.1 and Cloudera Manager 5.8.

 

In Yarn tuning Excel, I see a field called "Physical Cores to Vcores Multiplier" which has the following description:

 

"Set this ratio based on the expected number of concurrent threads per core. Use 1 for CPU intensive tasks up to 4 for standard I/O bound tasks."

 

I could not find a Cloudera Manager setting corresponding to it. How do we tell Yarn that it can use a multiplier of 4 ? Is there a default value to it ?

 

Or is this multiplier indicated to Yarn by setting "yarn.nodemanager.resource.cpu-vcores" parameter to a desired value instead of just the count of physical core ?

 

Say, if I have 24 physical cores, and I intend to use 50% of it for Yarn, then should I put 12 for this parameter to indicate that I want a ratio of 1:1 and put 48 to indicate a ratio of 1:4 ?

 

We use static pool resource allocation for various Hadoop components (Hbase, Solr, Hadoop, Yarn) and we set ~50% of resources to Yarn. But I could not find the multiplier anywhere.

 

Thanks,

Vivek

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Yarn Physical to Virtual Core multiplier

Super Collaborator

The "multiplier" is not a parameter.

You directly set the number of vcpu you want yarn to use. I guess you have to do the math yourself before setting the value.

4 REPLIES 4

Re: Yarn Physical to Virtual Core multiplier

Super Collaborator

The "multiplier" is not a parameter.

You directly set the number of vcpu you want yarn to use. I guess you have to do the math yourself before setting the value.

Re: Yarn Physical to Virtual Core multiplier

Explorer

Thank you Mathieu.d

 

Is this the vcpu parameter you are referring to ? yarn.nodemanager.resource.cpu-vcores

 

Thanks,

Vivek

Highlighted

Re: Yarn Physical to Virtual Core multiplier

Super Collaborator

Yes, this one.

Re: Yarn Physical to Virtual Core multiplier

Explorer
Thanks for confirming. I appreciate your quick feedback.
Don't have an account?
Coming from Hortonworks? Activate your account here