what's the recommended value of mapreduce.job.max.split.locations ?

[#1]

and what's the exact purpose of this? I understood the default value (10) is very low and that we should put that to cluster nodes number, what would be the impact of setting it to, say 1000 or even 100000?

thanks experts !

[#2 · Accepted solution]

This configuration is involved since MR v1. It serves as an up limit for DN locations of job split which intend to protect the JobTracker from overloaded by jobs with huge numbers of split locations. For YARN in Hadoop 2, this concern is lessened as we have per job AM instead of JT. However, it will still impact RM as RM will potentially see heavy request from the AM which tries to obtain many localities for the split. With hitting this limit, it will truncate location number to given limit with sacrifice a bit data locality but get rid of the risk to hit bottleneck of RM.

Depends on your job's priority (I believer it is a per job configuration now), you can leave it as a default (for lower or normal priority job) or increase to a larger number. Increase this value to larger than DN number will be the same impact as set it to DN's number.

[#3]

thanks @Junping Du