question Re: Write MR job output to existing directory in Archives of Support Questions (Read Only)

Write MR job output to existing directory

shubham2629 — Wed, 06 Apr 2016 13:27:05 GMT

Hello,

Is it possible to write MR job output to existing directory without deleting it(incremental write)?

Thanks

Shubham

Re: Write MR job output to existing directory

Former Member — Wed, 06 Apr 2016 13:49:57 GMT

You can write the following kind of logic in your MR application to make sure that if the directory exist already then delete it first.

Configuration conf = new Configuration(); 
FileSystem fs = FileSystem.get(conf); 
if(fs.exists(new Path(args[1]))) {
      /*If exist delete the output path*/
       fs.delete(new Path(args[1]),true); 
}

Re: Write MR job output to existing directory

shubham2629 — Wed, 06 Apr 2016 14:07:54 GMT

I do not want to delete existing directory as i mentioned above. I want to write more data to existing directory.

Re: Write MR job output to existing directory

pminovic — Wed, 06 Apr 2016 14:33:37 GMT

It can be done by extending the OutputFormat class and overwriting the OutputFormat.checkOutputSpecs method so that it doesn't throw exception when the output path already exists. After that, register the new class using JobConf.setOutputFormatClass method [some more details here].

Re: Write MR job output to existing directory

shubham2629 — Wed, 06 Apr 2016 15:42:00 GMT

thanks!. I will try that.

Re: Write MR job output to existing directory

shubham2629 — Tue, 12 Apr 2016 00:24:32 GMT

It worked. Thanks!