Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is there a way that i can move the outputs to subfolder of output path.?

SOLVED Go to solution

Is there a way that i can move the outputs to subfolder of output path.?

Explorer

Hi,

 

I am trying a solution, in which i have to take files recursively from the folder and perform few calculations and send the required data to output folder.

 

There is little dependency on the date of the file. I am using multipleoutputs for now, to have that date in file name.

But, it will be very good, if i there is a chance of creating sub folders in the output path with dates and move the respective files into that folder.

 

 

One way is to use FileSystem and create required directory and write the file output, without any output from mapreduce.

But, is there a way i can make mapreduce to write into required sub folder.? might not be possible, but just want to give a thought.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Is there a way that i can move the outputs to subfolder of output path.

Master Guru
In MultipleOutput's write(K, V, String) method, you can pass a relative
path for the baseOutputPath (last) argument such as
"my-sub-dir/my-filename-prefix". This will allow you to produce
subdirectories under the job output directory.

Please ensure to use only relative paths, and not absolute ones, as
otherwise your output paths will not be respected by the
FileOutputCommitter process.
1 REPLY 1

Re: Is there a way that i can move the outputs to subfolder of output path.

Master Guru
In MultipleOutput's write(K, V, String) method, you can pass a relative
path for the baseOutputPath (last) argument such as
"my-sub-dir/my-filename-prefix". This will allow you to produce
subdirectories under the job output directory.

Please ensure to use only relative paths, and not absolute ones, as
otherwise your output paths will not be respected by the
FileOutputCommitter process.