Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
Go to solution
Question about map output
Labels:
- Labels:
-
Apache Hadoop
-
HDFS
Contributor
Created on ‎09-06-2014 10:50 AM - edited ‎09-16-2022 02:07 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm looking for clarification on something I've read in Hadoop: The Definitive Guide. It states that map output is local and not in HDFS. But the map task runs on the node where the data resides (usually) and that is HDFS, correct? Or is it the case that the I/O done by the map is standard Java I/O and not something like hadoop fs -put?
Thanks in advance to all who answer.
Thanks in advance to all who reply.
Kevin
Kevin
1 ACCEPTED SOLUTION
Guru
Created ‎09-08-2014 12:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The map task's local output is not stored within HDFS, rather in temporary
directories on that specific node (see property mapreduce.cluster.local.dir)
written using standard file I/O
https://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-de...
directories on that specific node (see property mapreduce.cluster.local.dir)
written using standard file I/O
https://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-de...
Regards,
Gautam Gopalakrishnan
Gautam Gopalakrishnan
1 REPLY 1
Guru
Created ‎09-08-2014 12:14 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The map task's local output is not stored within HDFS, rather in temporary
directories on that specific node (see property mapreduce.cluster.local.dir)
written using standard file I/O
https://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-de...
directories on that specific node (see property mapreduce.cluster.local.dir)
written using standard file I/O
https://hadoop.apache.org/docs/r2.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-de...
Regards,
Gautam Gopalakrishnan
Gautam Gopalakrishnan
