Support Questions
Find answers, ask questions, and share your expertise

HDFS balancer with small files

Explorer

Hi, 

I have an HDFS cluster with 3 nodes. The cluster holds lots of small files (KB) and I have reached Millions of blocks per node.

 

I have added 4 more new servers to the cluster and started the balancer process but it looks that it does not do much. - The goal is to reduce the Million of blocks per server 

 

In order to balance the small-size files should i change the value of the following parameter to support moving files from 1KB size?

Ddfs.balancer.getBlocks.min-block-size=1048

2 REPLIES 2

Rising Star

Hi @dzbeda,

The definition of "dfs.balancer.getBlocks.min-block-size" is "Smallest block to consider for moving".

What is the version of hadoop? Is it CDH or HDP? What is the version of CDH / HDP?

For CDH please refer to: 

https://docs.cloudera.com/documentation/enterprise/latest/topics/admin_hdfs_balancer.html#cmug_topic...

https://docs.cloudera.com/documentation/enterprise/6/properties/6.1/topics/cm_props_cdh5160_hdfs.htm...

HDFS Balancer and DataNode Space Usage Considerations:

https://my.cloudera.com/knowledge/HDFS-Balancer-and-DataNode-Space-Usage-Considerations?id=73869

 

Regards,

Will

Community Manager

@dzbeda, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
; ;