question Re: Recommended config: mapreduce.input.fileinputformat.split.[minsize|maxsize] in Archives of Support Questions (Read Only)

Recommended config: mapreduce.input.fileinputformat.split.[minsize|maxsize]

jstraub — Tue, 27 Oct 2015 15:55:33 GMT

What is the purpose of the following two configuration parameters in mapred-size.xml? What are recommended values?

mapreduce.input.fileinputformat.split.minsize
mapreduce.input.fileinputformat.split.maxsize

Thanks 🙂

Re: Recommended config: mapreduce.input.fileinputformat.split.[minsize|maxsize]

nsabharwal — Tue, 27 Oct 2015 19:41:42 GMT

@Jonas Straub

I found this really useful

Also, from Apache doc

Deprecated property name

mapred.min.split.size

New

mapreduce.input.fileinputformat.split.minsize

Re: Recommended config: mapreduce.input.fileinputformat.split.[minsize|maxsize]

jstraub — Wed, 28 Oct 2015 18:16:47 GMT

Thanks @Neeraj

I also found these two books:

Pro Apache Hadoop

Hadoop Definitive Guide

And both are basically saying that mapreduce.input.fileinputformat.split.minsize < dfs.blocksize < ...maxsize

Smartsense recommended: 105MB (minsize) and 270MB (maxsize)

Our current block setting is 64MB, although Smartsense recommended 128MB blocksize, so it kind of fits the min/max recommendations as well as the descriptions from the books.

Re: Recommended config: mapreduce.input.fileinputformat.split.[minsize|maxsize]

nsabharwal — Wed, 28 Oct 2015 18:28:23 GMT

Thanks for sharing 🙂 @Jonas Straub