Reply
Explorer
Posts: 38
Registered: ‎09-29-2015

Spliting resources among service

Hi All, 

 

Recently I deployed 4 node cluster with CDH 5.7.1 and hardware details are as follows :

 

RAM - 16Gb , Hard disk - 500Gb

 

Can someone tell me how to allocate resources to services, mostly I am using hive, spark and Hbase (I/O intense jobs).

 

Is there any pre-defined calculations on how to split resources ? Please share. 

 

Just FyI : I saw cloudera performance tuning sheet and it didn't help me much 

 

Thanks

Kishore 

Posts: 62
Topics: 0
Kudos: 14
Blog Posts: 0
Ideas: 0
Solutions: 6
Registered: ‎11-26-2015

Re: Spliting resources among service

Dear Kishore,

 

I think static service pools what you are looking for. This documentation can help you:

 

http://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_service_pools.html

 

Best regards,

 

    Gabor

 

 

Explorer
Posts: 38
Registered: ‎09-29-2015

Re: Spliting resources among service

Hi Gabor, 

 

Thanks for the reply, I will go through this link. To make my request simply. I want to assgin values for Yarn parameters like :

 

yarn.scheduler.maximum-allocation-mb, yarn.nodemanager.resource.memory-mb, yarn.app.mapreduce.am.resource.cpu-vcores , yarn.app.mapreduce.am.resource.mb, yarn.app.mapreduce.am.resource.mb, etc .. 

 

One more thing, I am confused about parameter Heap to Container Size Ratio (in my case it is 0.8).

 

Thanks

Kishore

Posts: 62
Topics: 0
Kudos: 14
Blog Posts: 0
Ideas: 0
Solutions: 6
Registered: ‎11-26-2015

Re: Spliting resources among service

Hello Kishore,

 

Here is a YARN tuning guide/spreadsheet which can help you to configure these parameters:

 

https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html

 

Gabor