question how to set number of map and reduce tasks in Archives of Support Questions (Read Only)

how to set number of map and reduce tasks

tajinderdhanjal — Wed, 01 Jun 2016 20:13:18 GMT

I am running a query which runs 52 map jobs simultaneously. Due to this my Resource manager container gets filled up completely and consumed up 100%. The query stucks at that point and giving no result.

I want to reduce number of map tasks which runs in parallel.

Re: how to set number of map and reduce tasks

PranayV — Wed, 01 Jun 2016 20:26:10 GMT

You will have to pass runtime parameters to Hive query.

See this post.

https://community.hortonworks.com/questions/905/how-are-number-of-mappers-determined-for-a-query-w.html

Re: how to set number of map and reduce tasks

grajagopal — Wed, 01 Jun 2016 21:15:42 GMT

@Tajinderpal Singh Also, look at mapreduce.job.reduce.slowstart.completedmaps properties in map-reduce and set this to 0.9. This property will ensure reducers not coming in early and waiting for mappers to complete there by avoiding hung jobs.

Re: how to set number of map and reduce tasks

sbomma — Thu, 02 Jun 2016 01:43:05 GMT

You can set the run time parameters within hive shell, or pass them through your script, as mentioned by Pranay. Also if you are using Tez, this particular article on how to tune the performance may come in Handy.

https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html