Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to append files to HDFS with Java; "current leaseholder is trying to recreate file" errors

How to append files to HDFS with Java; "current leaseholder is trying to recreate file" errors

Explorer

I'm having trouble figuring out a safe way to append to files in HDFS.

 

I'm using a small, 3-node Hadoop cluster (CDH v.5.3.9 to be specific).

 

Our process is a data pipeliner which is multi-threaded (8 threads) and it has a stage which appends lines of delimited text to files in a dedicated directory on HDFS. I'm using locks to synchronize access of the threads to the buffered writers which append the data.

 

My first issue is deciding on the approach generally.

 

Approach A is to open the file, append to it, then close it for every line appended. This seems slow and would seem to create too many small blocks, or at least I see some such sentiment in various posts.

 

Approach B is to cache the writers but periodically refresh them to make sure the list of writers doesn't grow unbounded (currently, it's one writer per each input file processed by the pipeliner). This seems like a more efficient approach but I imagine having open streams over a period of time however controlled may be an issue, especially for output file readers (?)

Beyond this, my real issues are two. I am using the FileSystem Java Hadoop API to do the appending and am intermittently getting these 2 exceptions:

 

org.apache.hadoop.ipc.RemoteException: failed to create file /output/acme_20160524_1.txt for DFSClient_NONMAPREDUCE_271210261_1 for client XXX.XX.XXX.XX because current leaseholder is trying to recreate file.

 

org.apache.hadoop.ipc.RemoteException: BP-1999982165-XXX.XX.XXX.XX-1463070000410:blk_1073760252_54540 does not exist or is not under Constructionblk_1073760252_545

40{blockUCState=UNDER_RECOVERY, primaryNodeIndex=1, replicas=[ReplicaUnderConstruction[[DISK]DS-ccdf4e55-234b-4e17-955f-daaed1afdd92:NORMAL|RBW], ReplicaUnderConst

ruction[[DISK]DS-1f66db61-759f-4c5d-bb3b-f78c260e338f:NORMAL|RBW]]}

 

Anyone have any ideas on either of those?

 

For the first problem, I've tried instrumenting logic discussed in this post: http://stackoverflow.com/questions/23833318/crashed-hdfs-client-how-to-close-remaining-open-files, which didn't seem to help.

 

I'm also interested in the role of the dfs.support.append property, if at all applicable.

 

My code for getting the file system:

userGroupInfo = UserGroupInformation.createRemoteUser("hdfs"); Configuration conf = new Configuration();

conf.set(key1, val1);

...

conf.set(keyN, valN);

fileSystem = userGroupInfo.doAs(new PrivilegedExceptionAction<FileSystem>() {

  public FileSystem run() throws Exception {

   return FileSystem.get(conf);

  }

});

 

My code for getting the OutputStream:

org.apache.hadoop.fs.path.Path file = ...

public OutputStream getOutputStream(boolean append) throws IOException {   

  OutputStream os = null;

  synchronized (file) {

    if (isFile()) {

      os = (append) ? fs.append(file) : fs.create(file, true);

    } else if (append) {

      // Create the file first, to avoid "failed to append to non-existent file" exception

      FSDataOutputStream dos = fs.create(file);

      dos.close();

      // or, this can be: fs.createNewFile(file);

      os = fs.append(file);

    }

    // Creating a new file

    else {

      os = fs.create(file);

    }

  }

  return os;

}

 

7 REPLIES 7

Re: How to append files to HDFS with Java; "current leaseholder is trying to recreate file&quot

Explorer

I got file appending working with CDH 5.3 / HDFS 2.5.0. My conclusions so far are as follows:

 

  • Cannot have one dedicated thread doing appends per file, or multiple threads writing to multiple files, whether we’re writing data via one and the same instance of the HDFS API FileSystem, or different instances.
  • Cannot refresh (i.e. close and reopen) the writers; they must stay open.
  • This last item leads to occasional relatively rare ClosedChannelException which appears to be recoverable (by retrying to append).
  • We use a single thread executor service with a blocking queue (one for appending to all files); a writer per file, the writers stay open (till the end of processing when they’re closed).
  • When we upgrade to CDH newer than 5.3, we’ll want to revisit this and see what threading strategy makes sense: one and only thread, one thread per file, multiple threads writing to multiple files. Additionally, we’ll want to see if writers can be/need to be periodically closed and reopened.
  • In addition, I have seen the following error as well, and was able to make it go away by setting 'dfs.client.block.write.replace-datanode-on-failure.policy' to 'NEVER' on the client side.

java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[XXX.XX.XXX.XX:50010, XXX.XX.XXX.XX:50010], original=[XXX.XX.XXX.XX:50010, XXX.XX.XXX.XX:50010]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969) ~[hadoop-hdfs-2.5.0.jar:?] at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035) ~[hadoop-hdfs-2.5.0.jar:?] at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184) ~[hadoop-hdfs-2.5.0.jar:?] at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:532) ~[hadoop-hdfs-2.5.0.jar:?]

Re: How to append files to HDFS with Java; "current leaseholder is trying to recreate file&

Master Guru
Unless your writing process is actually distributed (vs. just parallel threads inside a single app) you may find it better to keep the files open (cache the OutputStream for the required maximum periods) instead of reopening for appends between parallel writer threads.

Append is an expensive operation in HDFS, and unless your use-case absolutely demands a multi-writer approach its not worth using it.

Re: How to append files to HDFS with Java; "current leaseholder is trying to recreate file&

Explorer

Thanks, Harsh.

 

Part of the issue was locating enough documentation to ascertain what the "best practice" for appends to files in HDFS is. The other part of it is understanding the support for appends in HDFS across the multiple CDH/Hadoop releases. Neither of these items is entirely clear unless I just haven't found a good writeup or documentation page on this. Hopefully my post will help.

 

Per your comments, yes, in this case we've got a single app with parallel threads. So far it's shaping up as having to use one and only one thread writing to all files, and keeping all buffered writers open. The issue here is that the app will be running continuously and we don't have a clear sign of when a given input file is done, so the map of file names to respective open writers will grow over time.

 

Additionally, your comment brings up a good point. We will be running on Spark as part of Spark jobs so this will be distributed. How does that affect HDFS file appends?

 

Will things be different once we upgrade CDH from 5.3 to 5.7?

Re: How to append files to HDFS with Java; "current leaseholder is trying to recreate file&

Explorer

Any word on any changes in CDH past version 5.3 as far as file appending is concerned? What about the issue of HDFS holding onto a lease for HDFS client X on a file and not letting another client Y append to the file?

Re: How to append files to HDFS with Java; "current leaseholder is trying to recreate file&

Explorer

Hi Harsh,

We are working with CDH5.7 security cluster.

We have a requirement that is to use 12 thread to append file to 12 hdfs files every 5 seconds.  Maybe there is  append more small log files of total 400~600M size every hour and we append file using outputstream open or close.

So, I would like to know what is the risk? and why do you say Append is an expensive operation in HDFS?

 

Would you please give me some advise? 
BR
Paul

Re: How to append files to HDFS with Java; "current leaseholder is trying to recreate file&

Master Guru
If your application is a long running one, then I'd advise keeping the file open for extended duration of time vs. closing and then re-opening for append.

Flume does this for example - it keeps files open for the entire rolling duration or size condition, even if no current data is being streamed in.

You can limit close+append to only the situation of a late delivery, for example.

Re: How to append files to HDFS with Java; "current leaseholder is trying to recreate file&

Explorer

Hi Harsh,

 

I got it.

Thank you for your excellent work.

 

BR

Paul

Don't have an account?
Coming from Hortonworks? Activate your account here