Support Questions
Find answers, ask questions, and share your expertise

how to set number of map and reduce tasks

Solved Go to solution

how to set number of map and reduce tasks

I am running a query which runs 52 map jobs simultaneously. Due to this my Resource manager container gets filled up completely and consumed up 100%. The query stucks at that point and giving no result.

I want to reduce number of map tasks which runs in parallel.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to set number of map and reduce tasks

Expert Contributor
3 REPLIES 3

Re: how to set number of map and reduce tasks

Expert Contributor

Re: how to set number of map and reduce tasks

Rising Star

@Tajinderpal Singh Also, look at mapreduce.job.reduce.slowstart.completedmaps properties in map-reduce and set this to 0.9. This property will ensure reducers not coming in early and waiting for mappers to complete there by avoiding hung jobs.

Re: how to set number of map and reduce tasks

Expert Contributor

You can set the run time parameters within hive shell, or pass them through your script, as mentioned by Pranay. Also if you are using Tez, this particular article on how to tune the performance may come in Handy.

https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer....