- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
(JAVA, MAP REDUCE) Read a File twice with Different Parameters
Created ‎07-01-2016 04:09 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I know that you can spawn two mappers of the same file if you type the addinputpath function twice with the same path, but I'd like the file to be processed slightly different each time.
Specifically, I want each time to use different parameters that I passed through the Job class (with configuration.set/get). When the files are different I get the path/name of the file by using the context/inputsplit classes to achieve that, but now that they are the same I can't differentiate them. Any thoughts?
Each mapper is a different maptask but i have no idea if i can use any info regarding the maptasks. Also I don't know the order the framework matches inputsplits to maptasks - it could be useful.
Alternatively I could duplicate the file(using a different name), but that would be a waste of resources
Created ‎07-01-2016 04:16 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
unique in that you need to process the same input 2x but with different
params each time. It seems a bit redundant to me given that you can do it
in a single task run vs. 2x the I/O cost…
But I believe the way you can solve your identifier problem is by writing
your own InputFormat wrapper over the existing InputFormat, which generates
special types of InputSplit objects (wrapper over regular FileSplit
classes). These input splits need to add in your identifiers as an extra
field, and you can extract and cast the same from your
context.getInputSplit() in the map-end to then differentiate the input.
Created ‎07-01-2016 04:16 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
unique in that you need to process the same input 2x but with different
params each time. It seems a bit redundant to me given that you can do it
in a single task run vs. 2x the I/O cost…
But I believe the way you can solve your identifier problem is by writing
your own InputFormat wrapper over the existing InputFormat, which generates
special types of InputSplit objects (wrapper over regular FileSplit
classes). These input splits need to add in your identifiers as an extra
field, and you can extract and cast the same from your
context.getInputSplit() in the map-end to then differentiate the input.
