reducer tasks long time

[#1]

Hi: what can i do to improve the time for the reducer???

I have 107 mapper and just 1 reduce, so, which parameters could i change??

maybe thoste?

mapreduce.job.counters.max

Thanks

[#2]

@Roberto Sancho

See this thread http://stackoverflow.com/questions/16836770/last-reducer-tasks-are-taking-long-time-to-finish

[#3]

@Roberto Sancho

Do you have support contract ?

Please install smartsense for better utilization of your cluster .

[#4]

Hi, we dont have yet, but the smartsense is free??

Thanks

[#5]

@Roberto Sancho It's part of HDP and for supported customers. http://hortonworks.com/blog/introducing-hortonworks-smartsense/

[#6]

Its mapred.reduce.tasks, if you run a mapreduce program from the hadoop client you would set it like this:

-Dmapred.reduce.tasks=x

Pig and Hive have different ways to predict reducer numbers.

[#7]

this -Dmapred.reduce.tasks=x is for mapreduce1 iam using mapreduce2 and yarn and i dont know how to change this parameter.

anny suggestion??

Thanks

[#8]

Still works on yarn, the official new one is mapreduce.job.reduces but I always used the one above and he still takes it.

[#9 · Accepted solution]

@Roberto Sancho

I would look at setting intermediate compression from map tasks and ouput compression from reduce tasks

You can also look at using combiner class.

mapreduce.map.output.compress
mapreduce.map.output.compress.codec

and output compression

mapreduce.output.fileoutputformat.compress.codec
mapreduce.output.fileoutputformat.compress.type
mapreduce.output.fileoutputformat.compress

[#10]

@Roberto Sancho here's a list of all deprecated mapred properties and new properties,

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

the property you're looking for is called mapreduce.job.reduces