Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

mapred.output.committer.class

avatar
Contributor

Could someone confirm the correct practice for setting the mapred.output.committer.class property? I was able to get it to work only if I set the property in my OutputFormat class. I tried

1) setting it in my SerDe class

2) putting it in mapred-site.xml

but in both cases there seems to be no effect. Thank you.

1 ACCEPTED SOLUTION

avatar
Master Guru

@Dayou Zhou

can you please set this property as well to make it work

mapreduce.use.directfileoutputcommitter=true

View solution in original post

4 REPLIES 4

avatar
Master Guru

@Dayou Zhou

can you please set this property as well to make it work

mapreduce.use.directfileoutputcommitter=true

avatar
Contributor

Hi @Sunile Manjee, thanks for the reply. I can try this but my understanding was that mapred.output.committer.class is for old API and mapreduce.use.directfileoutputcommitter is for new API, and it's an either-or. So would it really help to use both?

avatar
Master Guru

ok I understand. I have only seen it implemented like this:

committer = ReflectionUtils.newInstance(conf.getClass(
          "mapred.output.committer.class", DirectFileOutputCommitter.class,
          org.apache.hadoop.mapred.OutputCommitter.class), conf);
    }

avatar
Master Guru

please advise if this has answered your quesiton