https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124733#M22694 <P>thanks <A rel="user" href="https://community.cloudera.com/users/168/bleonhardi.html" nodeid="168">@Benjamin Leonhardi</A></P> Mon, 14 Mar 2016 17:26:29 GMT mhdeshmukh22 2016-03-14T17:26:29Z https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124730#M22691 Sun, 13 Mar 2016 23:23:22 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124730#M22691 mhdeshmukh22 2016-03-13T23:23:22Z https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124731#M22692 <P>You can't set number of mappers, it is determined by number of blocks in your dataset. </P><P>The number of maps is usually driven by the total size of the inputs, that is, the total number of blocks of the input files.</P><P>The right level of parallelism for maps seems to be around 10-100 maps per-node, although it has been set up to 300 maps for very cpu-light map tasks. Task setup takes a while, so it is best if the maps take at least a minute to execute.</P><P>Thus, if you expect 10TB of input data and have a blocksize of 128MB, you’ll end up with 82,000 maps, unless Configuration.set(MRJobConfig.NUM_MAPS, int) (which only provides a hint to the framework) is used to set it even higher.</P><P><A href="https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Mapper" target="_blank">https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Mapper</A></P> Sun, 13 Mar 2016 23:29:38 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124731#M22692 aervits 2016-03-13T23:29:38Z https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124732#M22693 <DIV>There is actually a way to change the numbers of mappers in Pig. Pig uses a CombineFileInputFormat to merge small files into bigger map tasks. This is enabled by default and can be modified with the following parameters: For the rest what Artem said.</DIV><UL> <LI>pig.maxCombinedSplitSize – Specifies the size, in bytes, of data to be processed by a single map. Smaller files are combined untill this size is reached. </LI><LI>pig.splitCombination – Turns combine split files on or off (set to “true” by default).</LI></UL> Mon, 14 Mar 2016 04:50:55 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124732#M22693 bleonhardi 2016-03-14T04:50:55Z https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124733#M22694 <P>thanks <A rel="user" href="https://community.cloudera.com/users/168/bleonhardi.html" nodeid="168">@Benjamin Leonhardi</A></P> Mon, 14 Mar 2016 17:26:29 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124733#M22694 mhdeshmukh22 2016-03-14T17:26:29Z https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124734#M22695 <P>thanks@Artem</P> Mon, 14 Mar 2016 17:26:59 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/We-can-set-the-number-of-reduce-tasks-for-the-MapReduce-jobs/m-p/124734#M22695 mhdeshmukh22 2016-03-14T17:26:59Z