Is there a way that i can move the outputs to subfolder of output path.?

Srini_D — Fri, 16 Sep 2022 09:34:51 GMT

Hi,

I am trying a solution, in which i have to take files recursively from the folder and perform few calculations and send the required data to output folder.

There is little dependency on the date of the file. I am using multipleoutputs for now, to have that date in file name.

But, it will be very good, if i there is a chance of creating sub folders in the output path with dates and move the respective files into that folder.

One way is to use FileSystem and create required directory and write the file output, without any output from mapreduce.

But, is there a way i can make mapreduce to write into required sub folder.? might not be possible, but just want to give a thought.

Re: Is there a way that i can move the outputs to subfolder of output path.

Harsh J — Tue, 21 Jul 2015 06:05:02 GMT

In MultipleOutput's write(K, V, String) method, you can pass a relative
path for the baseOutputPath (last) argument such as
"my-sub-dir/my-filename-prefix". This will allow you to produce
subdirectories under the job output directory.

Please ensure to use only relative paths, and not absolute ones, as
otherwise your output paths will not be respected by the
FileOutputCommitter process.

question Is there a way that i can move the outputs to subfolder of output path.? in Archives of Support Questions (Read Only)

Is there a way that i can move the outputs to subfolder of output path.?

Re: Is there a way that i can move the outputs to subfolder of output path.