question Re: HDFS balancer with small files in Support Questions

HDFS balancer with small files

dzbeda — Sat, 16 Oct 2021 08:38:34 GMT

Hi,

I have an HDFS cluster with 3 nodes. The cluster holds lots of small files (KB) and I have reached Millions of blocks per node.

I have added 4 more new servers to the cluster and started the balancer process but it looks that it does not do much. - The goal is to reduce the Million of blocks per server

In order to balance the small-size files should i change the value of the following parameter to support moving files from 1KB size?

Ddfs.balancer.getBlocks.min-block-size=1048

Re: HDFS balancer with small files

willx — Sun, 17 Oct 2021 13:45:25 GMT

Hi @dzbeda,

The definition of "dfs.balancer.getBlocks.min-block-size" is "Smallest block to consider for moving".

What is the version of hadoop? Is it CDH or HDP? What is the version of CDH / HDP?

For CDH please refer to:

https://docs.cloudera.com/documentation/enterprise/latest/topics/admin_hdfs_balancer.html#cmug_topic_5_14__section_lqb_rzp_x2b

https://docs.cloudera.com/documentation/enterprise/6/properties/6.1/topics/cm_props_cdh5160_hdfs.html#concept_6.1.x_balancer_props

HDFS Balancer and DataNode Space Usage Considerations:

https://my.cloudera.com/knowledge/HDFS-Balancer-and-DataNode-Space-Usage-Considerations?id=73869

Regards,

Will

Re: HDFS balancer with small files

VidyaSargur — Thu, 21 Oct 2021 17:30:40 GMT

@dzbeda, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.