Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

can we limit no of reducers used by TEZ job ?

can we limit no of reducers used by TEZ job ?

New Contributor
 
2 REPLIES 2

Re: can we limit no of reducers used by TEZ job ?

Yes.. I will write an article on this but for now you control this using the following parameter:

hive.exec.reducers.bytes.per.reducer

If you decrease it, you get more reducers.

If you increase it, you get less. By default it is 1 GB.

Try starting from 256MB and see the amount of reducers created.

Link to article

https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer....

Re: can we limit no of reducers used by TEZ job ?

In addition to what Ancil wrote you can also simply set the number of reducers as well:

set mapred.reduce.tasks=xxx;

The Hive guys don't like it too much because it can obviously result in bad performance if you don't know what you are doing. But in edge cases ... I use it for example to determine the number of ORC files during big loads.

http://www.slideshare.net/BenjaminLeonhardi/hive-loading-data

Don't have an account?
Coming from Hortonworks? Activate your account here