Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Reducers failling due to Error: java.io.IOException: File already exists:s3n://basket1/part-r-00016.deflate

Highlighted

Reducers failling due to Error: java.io.IOException: File already exists:s3n://basket1/part-r-00016.deflate

Explorer

Hello,

I have set job output path to "s3://basket1". I am writing to already existing directory/basket.

Few reducers are failing due to following error even though deflate files do not exist before starting job.

Error:

App > 16/04/11 17:25:47 INFO mapreduce.Job: Task Id : attempt_1460385692451_0006_r_000016_2, Status : FAILED App > Error: java.io.IOException: File already exists:s3n://basket1/part-r-00016.deflate App > at org.apache.hadoop.fs.s3native.NativeS3FileSystem.create(NativeS3FileSystem.java:813) App > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:914) App > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:895) App > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:792) App > at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:135) App > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540) App > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614) App > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) App > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:165) App > at java.security.AccessController.doPrivileged(Native Method) App > at javax.security.auth.Subject.doAs(Subject.java:415) App > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1635) App > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:160)

-Shubham

2 REPLIES 2
Highlighted

Re: Reducers failling due to Error: java.io.IOException: File already exists:s3n://basket1/part-r-00016.deflate

s3n and some of the object stores (swift://, s3a://) are not filesystems. This is something covered in the hadoop docs. Essentially, operations we'd treat as atomic and fast in a filesystem (creating a file with an existence check, renaming directories) are, against object stores, slow, prone to race conditions and can fail in unusual ways. This isn't unique to MapReduce either; Spark has just pulled its "DirectCommitter" because while it was fast against object stores, it ran the risk of corruption.

My recommendations (as I also give to colleagues)

  1. Don't use an object store as a destination of work
  2. instead: output to HDFS and then copy
  3. If you really use S3 as a destination, (a) use s3a:// (from Hadoop 2.7 and HDP 2.3), (b) turn off speculation. Speculative execution will go horribly wrong.
  4. take a look at Netflix's s3mper. They use amazon dynamo to store the directory information, so can implement the behaviours we expect from a filesystem.

Re: Reducers failling due to Error: java.io.IOException: File already exists:s3n://basket1/part-r-00016.deflate

Explorer

Thanks for the recommendations!. We need destination as s3 for the job. I tried different options. Job worked fine when disabled s3 multipart upload with DFOC enabled. We need DFOC enabled otherwise it takes long time for reducers to write o/p to S3. We are looking other options as well.

Don't have an account?
Coming from Hortonworks? Activate your account here