HDFS Block size 1Gb/2GB

sbpothineni — Thu, 06 Sep 2018 17:50:52 GMT

CDH enterprise 5.14.0

I am trying to use larger block size like 1GB/2GB.

In our case the files are 5GB to 14GB size and we process whole file per mapper,

is there any side effects to using larger block size like 1GB, 2GB? Like HDFS stability when doing replication?

Re: HDFS Block size 1Gb/2GB

Fawze — Thu, 06 Sep 2018 19:37:55 GMT

Hi @sbpothineni Why not using CombineFileInputFormat?

Re: HDFS Block size 1Gb/2GB

Harsh J — Fri, 07 Sep 2018 02:57:25 GMT

There are a few cons to raising your block size:

- Increased cost of recovery during write failures

When a client is writing a new block into the DataNode pipeline and one of the DataNode fails, there is a enabled-by-default recovery feature that will attempt to refill the gap in the replicated pipeline by transferring the partially written block from one of the remaining good DataNodes to a new DataNode.

When this happens, the client is blocked (the outstream,write(…) caller is blocked in the API code). With increased block size, the time waited will also increase greatly depending on how much of the partial block data was written before the failure occurred.

A worst-case wait example would involve the time required for network-copying 1.99 GiB for a 2 GiB block size because an involved DN may have failed at that specific point.

- Cost of replication caused by DataNode loss or decommission

When a DataNode is lost or is being decommissioned, the system has to react by re-filling the gaps in replica counts it creates. With smaller block sizes this activity is easy to spread randomly across the cluster, as several different nodes overall can take part in the re-replicate process. With larger blocks, only a few DNs can participate, and another consequence could be more lopsided space usage across DNs.

That said, use of 1-2 GiB is not unheard of and I've seen a few large clusters apply that as their default block size. Its just worth being aware of the cons, looking out for such impact and tuning accordingly as you go.

HDFS certainly functions at its best for large sized files, and your usage seems in accordance with that.

question Re: HDFS Block size 1Gb/2GB in Archives of Support Questions (Read Only)

HDFS Block size 1Gb/2GB

Re: HDFS Block size 1Gb/2GB

Re: HDFS Block size 1Gb/2GB