Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to Reduce the data volume during shuffling between Mapper and Reducer Node ?

How to Reduce the data volume during shuffling between Mapper and Reducer Node ?

New Contributor

How to Reduce the data volume during shuffling between Mapper and Reducer Node ?

2 REPLIES 2
Highlighted

Re: How to Reduce the data volume during shuffling between Mapper and Reducer Node ?

New Contributor

Improve the performance of data transfer between Mapper and Reducer is by using the Combiner function. Combiner works as a mini reducer which operates on data generated by Mapper and used for the purpose of optimization.

2nd option is we can compress the intermediate output generated by Mapper with the below command in driver class

Re: How to Reduce the data volume during shuffling between Mapper and Reducer Node ?

New Contributor

Hello @Harshali Patel, did you see my answer here?

I hope this helps.

Don't have an account?
Coming from Hortonworks? Activate your account here