Generally, the size of data blocks would be 128mb across all the Datanodes. But, if you have small files then you might see smaller blocks on some Datanodes as well. So Datanodes with different disk spaces would have uneven "Number of Blocks" and the Balancing happens based on the difference in the DFS usage and not by the difference in block count. ... View more

Hi @Ben1996, As this is very specific use case, I don't see any Cloudera doc which could cover this. The basic requirements would be to import the hdfs module and provide the HDFS configs.   For ex, https://tahiriamine9.medium.com/python-hdfs-cd822199799e#:~:text=Let's%20have%20an%20example%20of,into%20HDFS%20with%20CSV%20format. ... View more

@Noel_0317 Do you have rack awareness configured for the Datanodes?   Also, check for any disk-level issues on the datanode.   Try enabling Debug for block placement : log4j.logger.org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy=DEBUG log4j.logger.org.apache.hadoop.hdfs.protocol.BlockStoragePolicy=DEBUG ... View more

Summary : Rebalance option is greyed out in the Cloudera Manager under the HDFS -1 or Balancer Actions dropdown menu. Symptoms The Rebalance option is grayed out in the Cloudera Manager under the HDFS -1 or Balancer Actions menu. You have verified that they are no currently running Balancer operations under the HDFS-1->Balancer Instance->Commands tab. The HDFS Balancer menu option is normally greyed out if there is a active Balancer job running. Instructions To diagnose and workaround this issue, perform the following actions: Log into the Cloudera Manager SQL database and check the status of BALANCER in the Cloudera Manager database using the following command.  select CONFIGURED_STATUS, ROLE_TYPE FROM ROLES where ROLE_TYPE = "BALANCER" ;  The normal status is NA. If the status returned is BUSY then the status must be changed to NA using the following command.  update ROLES set CONFIGURED_STATUS="NA" where ROLE_TYPE = "BALANCER"; Re-execute the select statement to check to ensure the configured_status is now indicating the correct state. After changing the status the Rebalance option should no longer be greyed out in Cloudera Manager. ... View more