question Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster? in Archives of Support Questions (Read Only)

Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

raocm08 — Mon, 02 May 2016 20:36:37 GMT

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

sshimpi — Mon, 02 May 2016 20:44:45 GMT

If I am not wrong its not very simple to answer the question. No of mappers depends on your compute power [CPU and memory] and also on the no of containers [when using yarn].

1JVM corresponds to 1 Mapper usually.

Depending upon your compute you need to configure the MR memory settings so hat you can use MAXresources [ie. mappers and reducers]

pls refer below link for -

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/determine-hdp-memory-config.html

http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

https://cloudcelebrity.wordpress.com/2013/08/14/12-key-steps-to-keep-your-hadoop-cluster-running-strong-and-performing-optimum/

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

bleonhardi — Mon, 02 May 2016 20:51:10 GMT

I don't think there is any max number Most likely some theoretical int number that you wouldn't reach in any cluster. Or do you mean at the same time? In that case it would be

(RAM * Nodenumber) / *yarn.scheduler.minimum-allocation-mb

So if you have a 100 nodes and your nodes have 96GB yarn memory and your minimum allocation size ( and your map size ) is 2GB it would be 4800.

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

ledel — Mon, 02 May 2016 20:52:20 GMT

It depends on the memory you have on your cluster.

You have an amount of RAM allocated to YARN (yarn.nodemanager.resource.memory-mb) on a node, then each mapper has a size (in MR it's mapreduce.map.memory.mb), that gives you an idea (you'll also use memory for ApplicationMasters, Reducers,etc.)

Re: Hi, Can some one tell me how many max mappers we can use in a HDP2.3 cluster?

ajay_kumar — Tue, 03 May 2016 06:02:31 GMT

As others explained it depends on number of containers available on your cluster.