HDFS balancer with small files

[#1]

Hi, 

I have an HDFS cluster with 3 nodes. The cluster holds lots of small files (KB) and I have reached Millions of blocks per node.

I have added 4 more new servers to the cluster and started the balancer process but it looks that it does not do much. - The goal is to reduce the Million of blocks per server 

In order to balance the small-size files should i change the value of the following parameter to support moving files from 1KB size?

Ddfs.balancer.getBlocks.min-block-size=1048

[#2]

Hi @dzbeda,

The definition of "dfs.balancer.getBlocks.min-block-size" is "Smallest block to consider for moving".

What is the version of hadoop? Is it CDH or HDP? What is the version of CDH / HDP?

For CDH please refer to: 

https://docs.cloudera.com/documentation/enterprise/latest/topics/admin_hdfs_balancer.html#cmug_topic...

https://docs.cloudera.com/documentation/enterprise/6/properties/6.1/topics/cm_props_cdh5160_hdfs.htm...

HDFS Balancer and DataNode Space Usage Considerations:

https://my.cloudera.com/knowledge/HDFS-Balancer-and-DataNode-Space-Usage-Considerations?id=73869

Regards,

Will

[#3]

@dzbeda, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.