question Re: How match number of pipelines with number of the blocks in the file in Archives of Support Questions (Read Only)

How match number of pipelines with number of the blocks in the file

fil — Fri, 16 Sep 2022 09:26:17 GMT

Hi dear community!

i'll very appreciate if someone could tell how match number of pipelines with number of the blocks in the file?

i mean pipelines that are creating for writhig HDFS block (https://issues.apache.org/jira/secure/attachment/12445209/appendDesign3.pdf)

is number of pipelines equal of number of blocks in required file? In other words one pipeline created for one file block, right? Or one pipeline per file (multiple blocks)?

thanks!

Re: How match number of pipelines with number of the blocks in the file

Harsh J — Wed, 08 Apr 2015 12:49:48 GMT

A pipeline is created to write multiple replicas for every single
under-construction block. For every block's end, the pipeline is
completed and closed, and a subsequent new pipeline is opened for the
next block (if there is one). There's no notion of "number of
pipelines" given that its a sequential operation - why do you seek
such a number?

Re: How match number of pipelines with number of the blocks in the file

fil — Wed, 08 Apr 2015 12:55:24 GMT

thanks for your reply!

let me clarify my question:

does each file block has unique set of the Datanodes?

or all blocks writes in single set of datanodes?

thanks!

Re: How match number of pipelines with number of the blocks in the file

Harsh J — Wed, 08 Apr 2015 12:59:48 GMT

Every block's replica location allocation gets a different set of DNs.
They may all have one common
DataNode if the writer client is running on a host that also runs a
DataNode, but the other replicas will be randomly selected (within or
outside of racks, depending on topology).

Re: How match number of pipelines with number of the blocks in the file

fil — Wed, 08 Apr 2015 13:10:18 GMT

Thanks! Everything clear now!