Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to set number of map and reduce tasks

avatar
Contributor

I am running a query which runs 52 map jobs simultaneously. Due to this my Resource manager container gets filled up completely and consumed up 100%. The query stucks at that point and giving no result.

I want to reduce number of map tasks which runs in parallel.

1 ACCEPTED SOLUTION

avatar
Super Collaborator
3 REPLIES 3

avatar
Super Collaborator

You will have to pass runtime parameters to Hive query.

See this post.

https://community.hortonworks.com/questions/905/how-are-number-of-mappers-determined-for-a-query-w.h...

avatar
Expert Contributor

@Tajinderpal Singh Also, look at mapreduce.job.reduce.slowstart.completedmaps properties in map-reduce and set this to 0.9. This property will ensure reducers not coming in early and waiting for mappers to complete there by avoiding hung jobs.

avatar
Super Collaborator

You can set the run time parameters within hive shell, or pass them through your script, as mentioned by Pranay. Also if you are using Tez, this particular article on how to tune the performance may come in Handy.

https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer....