Reply
New Contributor
Posts: 9
Registered: ‎02-28-2017

Allocation of Datanode Memory and Nodemanger memory - Clarification Newbie

I am newbie to Cloudera Hadoop . Working on Mapreduce program.

I would like to increase the datanode memory ,  nodemanager memory ,

 

Should I have to increase in the hadoop-env.sh and Yarn-site.xml in the below tags 

I have been allocated around 128GB of RAM in my slave nodes.

 

So is there any base line to start with - more of a crap shot. My program is indeed a  heavy computation.

 

if I want to increase the datanode memory should i have to the put the numbers inside  HADOOP_DATANODE_OPTS  ? or is there any other tags ? please help me on this. 

 

yarn.nodemanager.resource.memory-mb
yarn.app.mapreduce.am.resource.mb
mapreduce.map.memory.mb
mapreduce.reduce.memory.mb	

any information is highly appreciable.

 

Thanks

Posts: 519
Topics: 14
Kudos: 92
Solutions: 45
Registered: ‎09-02-2016

Re: Allocation of Datanode Memory and Nodemanger memory - Clarification Newbie

New Contributor
Posts: 9
Registered: ‎02-28-2017

Re: Allocation of Datanode Memory and Nodemanger memory - Clarification Newbie

@saranvisa

 

Thanks for the response . Read your thread . In that post everything is pertain to Mapreduce . 

we have 128 GB allocated for Slave nodes. 

 

Could you please let me know what should go inside the base number to start with. 

yarn.nodemanager.resource.memory-mb

we have 2  hexa cores  runining . please let me know if this correct 

yarn.nodemanager.resource.cpu-vcores = 12 
yarn.scheduler.minimum-allocation-vcores =	1
yarn-scheduler.maximum-allocation-vcores =  10

what should go inside 

 

yarn.scheduler.minimum-allocation-mb	
yarn.scheduler.maximum-allocation-mb
Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: Allocation of Datanode Memory and Nodemanger memory - Clarification Newbie

Take a look here, it gives some good calcs to reach a baseline for the YARN/MR memory settings. Cores are straightforward leave a core or two for the other process and the rest to YARN containers.

All of these changes should be done through CM, if not using CM then in the yarn-site.xml and mapred-site.xml.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_installing_manually_book/content/determin...
Posts: 1,047
Kudos: 130
Solutions: 62
Registered: ‎04-06-2015

Re: Allocation of Datanode Memory and Nodemanger memory - Clarification Newbie

Here is one of our Community Knowledge articles that may also be of assistance when calculating memory size.

 

Selecting the Right Hardware for Your New Hadoop Cluster



Cy Jervis, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum