I have set job output path to "s3://basket1". I am writing to already existing directory/basket.
Few reducers are failing due to following error even though deflate files do not exist before starting job.
App > 16/04/11 17:25:47 INFO mapreduce.Job: Task Id : attempt_1460385692451_0006_r_000016_2, Status : FAILED App > Error: java.io.IOException: File already exists:s3n://basket1/part-r-00016.deflate App > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.create(NativeS3FileSystem.java:813) App > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:914) App > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:895) App > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:792) App > at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:135) App > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540) App > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614) App > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) App > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:165) App > at java.security.AccessController.doPrivileged(Native Method) App > at javax.security.auth.Subject.doAs(Subject.java:415) App > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1635) App > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:160)
s3n and some of the object stores (swift://, s3a://) are not filesystems. This is something covered in the hadoop docs. Essentially, operations we'd treat as atomic and fast in a filesystem (creating a file with an existence check, renaming directories) are, against object stores, slow, prone to race conditions and can fail in unusual ways. This isn't unique to MapReduce either; Spark has just pulled its "DirectCommitter" because while it was fast against object stores, it ran the risk of corruption.
My recommendations (as I also give to colleagues)
Thanks for the recommendations!. We need destination as s3 for the job. I tried different options. Job worked fine when disabled s3 multipart upload with DFOC enabled. We need DFOC enabled otherwise it takes long time for reducers to write o/p to S3. We are looking other options as well.