Reply
Explorer
Posts: 7
Registered: ‎08-15-2016
Accepted Solution

Yarn Physical to Virtual Core multiplier

[ Edited ]

We use CDH version 5.7.1 and Cloudera Manager 5.8.

 

In Yarn tuning Excel, I see a field called "Physical Cores to Vcores Multiplier" which has the following description:

 

"Set this ratio based on the expected number of concurrent threads per core. Use 1 for CPU intensive tasks up to 4 for standard I/O bound tasks."

 

I could not find a Cloudera Manager setting corresponding to it. How do we tell Yarn that it can use a multiplier of 4 ? Is there a default value to it ?

 

Or is this multiplier indicated to Yarn by setting "yarn.nodemanager.resource.cpu-vcores" parameter to a desired value instead of just the count of physical core ?

 

Say, if I have 24 physical cores, and I intend to use 50% of it for Yarn, then should I put 12 for this parameter to indicate that I want a ratio of 1:1 and put 48 to indicate a ratio of 1:4 ?

 

We use static pool resource allocation for various Hadoop components (Hbase, Solr, Hadoop, Yarn) and we set ~50% of resources to Yarn. But I could not find the multiplier anywhere.

 

Thanks,

Vivek

Posts: 177
Topics: 8
Kudos: 28
Solutions: 19
Registered: ‎07-16-2015

Re: Yarn Physical to Virtual Core multiplier

The "multiplier" is not a parameter.

You directly set the number of vcpu you want yarn to use. I guess you have to do the math yourself before setting the value.

Explorer
Posts: 7
Registered: ‎08-15-2016

Re: Yarn Physical to Virtual Core multiplier

Thank you Mathieu.d

 

Is this the vcpu parameter you are referring to ? yarn.nodemanager.resource.cpu-vcores

 

Thanks,

Vivek

Posts: 177
Topics: 8
Kudos: 28
Solutions: 19
Registered: ‎07-16-2015

Re: Yarn Physical to Virtual Core multiplier

Yes, this one.

Highlighted
Explorer
Posts: 7
Registered: ‎08-15-2016

Re: Yarn Physical to Virtual Core multiplier

Thanks for confirming. I appreciate your quick feedback.