question mapred.output.committer.class in Archives of Support Questions (Read Only)

mapred.output.committer.class

dz186001 — Thu, 21 Jul 2016 07:25:38 GMT

Could someone confirm the correct practice for setting the mapred.output.committer.class property? I was able to get it to work only if I set the property in my OutputFormat class. I tried

1) setting it in my SerDe class

2) putting it in mapred-site.xml

but in both cases there seems to be no effect. Thank you.

Re: mapred.output.committer.class

sunile_manjee — Thu, 21 Jul 2016 10:34:33 GMT

@Dayou Zhou

can you please set this property as well to make it work

mapreduce.use.directfileoutputcommitter=true

Re: mapred.output.committer.class

dz186001 — Thu, 21 Jul 2016 11:18:14 GMT

Hi @Sunile Manjee, thanks for the reply. I can try this but my understanding was that mapred.output.committer.class is for old API and mapreduce.use.directfileoutputcommitter is for new API, and it's an either-or. So would it really help to use both?

Re: mapred.output.committer.class

sunile_manjee — Thu, 21 Jul 2016 11:42:47 GMT

ok I understand. I have only seen it implemented like this:

committer = ReflectionUtils.newInstance(conf.getClass(
          "mapred.output.committer.class", DirectFileOutputCommitter.class,
          org.apache.hadoop.mapred.OutputCommitter.class), conf);
    }

Re: mapred.output.committer.class

sunile_manjee — Mon, 13 Mar 2017 09:14:01 GMT

please advise if this has answered your quesiton