@BrianChan  Cluster Average Utilization Calculation: The cluster average utilization during HDFS rebalancing is typically calculated based on the configured capacity of the cluster. The configured capacity represents the total storage capacity allocated to the HDFS cluster as defined in the cluster's configuration settings. Individual Utilization Calculation: Individual utilization during HDFS rebalancing is usually calculated based on the sum of DFS used and remaining space for each datanode. This calculation provides an accurate representation of how much storage is currently being utilized on each datanode and how much space is available for additional data storage. Difference in File Moving Size: The difference between the initially reported file moving size and the actual file moving size in the balancer log can occur due to various factors. These may include changes in data distribution across datanodes during the rebalancing process, optimizations performed by the balancer algorithm, or adjustments made based on real-time cluster conditions and performance considerations. Exceeding DataNode Balancing Bandwidth: While the datanode balancing bandwidth is configured to limit the amount of data transferred between datanodes per second during HDFS rebalancing, it's possible for the actual bandwidth consumption to exceed this limit under certain circumstances. Factors such as network congestion, variations in data transfer rates, or optimizations performed by the balancer algorithm can contribute to bandwidth consumption exceeding the configured limit.   Regards, Chethan YM ... View more

@Shivakuk When you replace a disk in an HDFS cluster, especially if it's a DataNode disk, the Hadoop system should handle data replication and rebalancing automatically. This means that once the new disk is added and the DataNode is back online, HDFS will redistribute the data across the cluster to maintain the configured replication factor. If data was wiped during or after the disk replacement process, it's critical to investigate why this occurred and take measures to prevent data loss in the future. Ensure that proper backup and recovery procedures are in place, and consider implementing data mirroring or replication to minimize the risk of data loss due to hardware failures. Regards, Chethan YM ... View more

@Timo, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.  ... View more

@Sokka, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.  ... View more

@MinhTruong, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  If you are still experiencing the issue, can you provide the information @ChethanYM has requested? ... View more

@Hanro As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks. ... View more

Hello @hebamahmoud  If the issue is has been from any of the above responses, Could you accept it as a solution? Regards, Chethan YM ... View more

@Yasine,  I'm happy to see you resolved your issue. Please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. ... View more

@tsunamisabella, If the reply helped resolve your issue, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.     ... View more

@tsunamisabella, did the responses from @ChethanYM  assist you in resolving your concerns? If so, could you kindly designate the most suitable response as the solution? This will be beneficial for other members who might come across a similar issue.   ... View more