question Re: How Map and Reduce operations are actually carried out in Archives of Support Questions (Read Only)

How Map and Reduce operations are actually carried out

Priyap — Tue, 11 Aug 2015 06:34:29 GMT

Hi,

I have gone through this question., can anyone pls tell me the correct answer with explanation?

Which best describes how TextInputFormat processes input files and line breaks?

A. Input file splits may cross line breaks. A line that crosses file splits is read by the RecordReader
of the split that contains the beginning of the broken line.
B. Input file splits may cross line breaks. A line that crosses file splits is read by the
RecordReaders of both splits containing the broken line.
C. The input file is split exactly at the line breaks, so each RecordReader will read a series of
complete lines.
D. Input file splits may cross line breaks. A line that crosses file splits is ignored.
E. Input file splits may cross line breaks. A line that crosses file splits is read by the RecordReader
of the split that contains the end of the broken line.

Thanks in advance

Re: How Map and Reduce operations are actually carried out

Harsh J — Tue, 11 Aug 2015 09:38:20 GMT

Please read http://wiki.apache.org/hadoop/HadoopMapReduce

Re: How Map and Reduce operations are actually carried out

Priyap — Tue, 11 Aug 2015 11:06:21 GMT

Hi Harish,

Thanks for the reply.
I had gone through the same link. But can you please explain me the same, I couldn't understand.

Re: How Map and Reduce operations are actually carried out

Harsh J — Tue, 11 Aug 2015 11:29:20 GMT

What part of it was not clear specifically? Could you quote, so it can be
explained further?

Re: How Map and Reduce operations are actually carried out

Priyap — Tue, 11 Aug 2015 11:31:41 GMT

" For example TextInputFormat will read the last line of the FileSplit past the split boundary and, when reading other than the first FileSplit, TextInputFormat ignores the content up to the first newline." .. what it means?

Re: How Map and Reduce operations are actually carried out

Harsh J — Tue, 11 Aug 2015 12:16:20 GMT

It just means that when the split offset (starting point) is 0, i.e. start
of file, we read the first line. Otherwise (non-zero offsets/starting
point/mid-points of file) we arbitrarily skip the first line, because we
know that the previous split's reader reads always an extra line at the end.

Does this help?

Re: How Map and Reduce operations are actually carried out

Priyap — Tue, 11 Aug 2015 12:32:41 GMT

Sorry harish, can you please explain in detail with small example.

Re: How Map and Reduce operations are actually carried out

Harsh J — Tue, 11 Aug 2015 13:17:20 GMT

Check http://stackoverflow.com/a/14540272 perhaps, which includes an
example.

Re: How Map and Reduce operations are actually carried out

Priyap — Tue, 18 Aug 2015 08:01:55 GMT

Thanks Harish