Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

reducer tasks long time

Solved Go to solution
Highlighted

reducer tasks long time

Super Collaborator

Hi: what can i do to improve the time for the reducer???

I have 107 mapper and just 1 reduce, so, which parameters could i change??

maybe thoste?

mapreduce.job.counters.max

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: reducer tasks long time

Mentor

@Roberto Sancho

I would look at setting intermediate compression from map tasks and ouput compression from reduce tasks

You can also look at using combiner class.

mapreduce.map.output.compress
mapreduce.map.output.compress.codec

and output compression

mapreduce.output.fileoutputformat.compress.codec
mapreduce.output.fileoutputformat.compress.type
mapreduce.output.fileoutputformat.compress
9 REPLIES 9

Re: reducer tasks long time

Re: reducer tasks long time

@Roberto Sancho

Do you have support contract ?

Please install smartsense for better utilization of your cluster .

Re: reducer tasks long time

Super Collaborator

Hi, we dont have yet, but the smartsense is free??

Thanks

Re: reducer tasks long time

Re: reducer tasks long time

Its mapred.reduce.tasks, if you run a mapreduce program from the hadoop client you would set it like this:

-Dmapred.reduce.tasks=x

Pig and Hive have different ways to predict reducer numbers.

Re: reducer tasks long time

Super Collaborator

this -Dmapred.reduce.tasks=x is for mapreduce1 iam using mapreduce2 and yarn and i dont know how to change this parameter.

anny suggestion??

Thanks

Re: reducer tasks long time

Still works on yarn, the official new one is mapreduce.job.reduces but I always used the one above and he still takes it.

Re: reducer tasks long time

Mentor

@Roberto Sancho

I would look at setting intermediate compression from map tasks and ouput compression from reduce tasks

You can also look at using combiner class.

mapreduce.map.output.compress
mapreduce.map.output.compress.codec

and output compression

mapreduce.output.fileoutputformat.compress.codec
mapreduce.output.fileoutputformat.compress.type
mapreduce.output.fileoutputformat.compress

Re: reducer tasks long time

Mentor

@Roberto Sancho here's a list of all deprecated mapred properties and new properties,

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

the property you're looking for is called mapreduce.job.reduces