question Re: During HDFS write mechanism - acknowledgement report is sent for each block written or only when all blocks are written? in Archives of Support Questions (Read Only)

During HDFS write mechanism - acknowledgement report is sent for each block written or only when all blocks are written?

clouderazone — Thu, 04 Feb 2016 05:15:21 GMT

In HDFS write, consider am writing a file of 1GB with 64MB as block size, then 16 blocks are created. I want to know whether, acknowledgement report is sent to Client for each block written or only when all blocks are written?

Re: During HDFS write mechanism - acknowledgement report is sent for each block written or only when all blocks are written?

nsabharwal — Thu, 04 Feb 2016 05:18:21 GMT

@Avinash C Namenode receives the confirmation as soon as block is written. very nicely explained here

Step 1: The client creates the file by calling create() method on DistributedFileSystem.

Step 2: DistributedFileSystem makes an RPC call to the namenode to create a new file in the filesystem’s namespace, with no blocks associated with it.

The namenode performs various checks to make sure the file doesn’t already exist and that the client has the right permissions to create the file. If these checks pass, the namenode makes a record of the new file; otherwise, file creation fails and the client is thrown an IOException. TheDistributedFileSystem returns an FSDataOutputStream for the client to start writing data to.

Re: During HDFS write mechanism - acknowledgement report is sent for each block written or only when all blocks are written?

ArpitAgarwal — Thu, 04 Feb 2016 07:28:39 GMT

Hi @Avinash C, the HDFS Architecture guide has a good description of the write pipeline (section 8.3.1.).

Re: During HDFS write mechanism - acknowledgement report is sent for each block written or only when all blocks are written?

clouderazone — Fri, 05 Feb 2016 03:44:37 GMT

Thanks Arpit. The section 8.3.1 was a very good read.

Re: During HDFS write mechanism - acknowledgement report is sent for each block written or only when all blocks are written?

clouderazone — Fri, 05 Feb 2016 03:44:52 GMT

Thanks Neeraj