Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Question about map output

Solved Go to solution
Highlighted

Question about map output

New Contributor

I'm looking for clarification on something I've read in Hadoop: The Definitive Guide.  It states that map output is local and not in HDFS.  But the map task runs on the node where the data resides (usually) and that is HDFS, correct?  Or is it the case that the I/O done by the map is standard Java I/O and not something like hadoop fs -put?

 

Thanks in advance to all who answer.

Thanks in advance to all who reply.

Kevin
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Question about map output

The map task's local output is not stored within HDFS, rather in temporary
directories on that specific node (see property mapreduce.cluster.local.dir)
written using standard file I/O

https://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-de...

Regards,
Gautam Gopalakrishnan
1 REPLY 1

Re: Question about map output

The map task's local output is not stored within HDFS, rather in temporary
directories on that specific node (see property mapreduce.cluster.local.dir)
written using standard file I/O

https://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-de...

Regards,
Gautam Gopalakrishnan