Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

Solved Go to solution

Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

New Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

Guru

It depends on the memory you have on your cluster.

You have an amount of RAM allocated to YARN (yarn.nodemanager.resource.memory-mb) on a node, then each mapper has a size (in MR it's mapreduce.map.memory.mb), that gives you an idea (you'll also use memory for ApplicationMasters, Reducers,etc.)

4 REPLIES 4

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

@Ram

If I am not wrong its not very simple to answer the question. No of mappers depends on your compute power [CPU and memory] and also on the no of containers [when using yarn].

1JVM corresponds to 1 Mapper usually.

Depending upon your compute you need to configure the MR memory settings so hat you can use MAXresources [ie. mappers and reducers]

pls refer below link for -

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/determi...

http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

https://cloudcelebrity.wordpress.com/2013/08/14/12-key-steps-to-keep-your-hadoop-cluster-running-str...

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

I don't think there is any max number Most likely some theoretical int number that you wouldn't reach in any cluster. Or do you mean at the same time? In that case it would be

(RAM * Nodenumber) / *yarn.scheduler.minimum-allocation-mb

So if you have a 100 nodes and your nodes have 96GB yarn memory and your minimum allocation size ( and your map size ) is 2GB it would be 4800.

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

Guru

It depends on the memory you have on your cluster.

You have an amount of RAM allocated to YARN (yarn.nodemanager.resource.memory-mb) on a node, then each mapper has a size (in MR it's mapreduce.map.memory.mb), that gives you an idea (you'll also use memory for ApplicationMasters, Reducers,etc.)

Highlighted

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

Rising Star

As others explained it depends on number of containers available on your cluster.

Don't have an account?
Coming from Hortonworks? Activate your account here