Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Write MR job output to existing directory

Solved Go to solution
Highlighted

Write MR job output to existing directory

Explorer

Hello,

Is it possible to write MR job output to existing directory without deleting it(incremental write)?

Thanks

Shubham

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Write MR job output to existing directory

It can be done by extending the OutputFormat class and overwriting the OutputFormat.checkOutputSpecs method so that it doesn't throw exception when the output path already exists. After that, register the new class using JobConf.setOutputFormatClass method [some more details here].

View solution in original post

5 REPLIES 5
Highlighted

Re: Write MR job output to existing directory

You can write the following kind of logic in your MR application to make sure that if the directory exist already then delete it first.

Configuration conf = new Configuration(); 
FileSystem fs = FileSystem.get(conf); 
if(fs.exists(new Path(args[1]))) {
      /*If exist delete the output path*/
       fs.delete(new Path(args[1]),true); 
}

Re: Write MR job output to existing directory

Explorer

I do not want to delete existing directory as i mentioned above. I want to write more data to existing directory.

Highlighted

Re: Write MR job output to existing directory

It can be done by extending the OutputFormat class and overwriting the OutputFormat.checkOutputSpecs method so that it doesn't throw exception when the output path already exists. After that, register the new class using JobConf.setOutputFormatClass method [some more details here].

View solution in original post

Highlighted

Re: Write MR job output to existing directory

Explorer

thanks!. I will try that.

Highlighted

Re: Write MR job output to existing directory

Explorer

It worked. Thanks!

Don't have an account?
Coming from Hortonworks? Activate your account here