Rack Awareness - Would modifying Rack Awareness topology trigger immediate BLOCK movement ?

Question

We have a Cluster running CDH 5.15.1 on 150+ nodes. Cluster nodes storage capacity varies (Mix of various node sizes, 10 TB, 13 TB, 25 TB, 48 TB).

Nodes are connected with a 10G network and presently running with a physical topology of Rack awareness.

Issue is, while few racks have 18 servers others are just having 1 or 2 servers on them. we have been advised to move away from this nonuniform rack awareness topology to a Logical Rack awareness topology which will help us put together UNIFORM no. of nodes on each rack.

I would like to understand, if the rack awareness topology is changed, would HDFS kick-off immediate Block movement or the new topology will only be effective for NEW blocks written on the cluster.

Anthony · Accepted Answer

Hi VikramD,

Thanks for reaching out. Regarding your question:

> If the rack awareness topology is changed, would HDFS kick-off immediate Block movement

> or the new topology will only be effective for NEW blocks written on the cluster?

The new topology will be applied when new blocks are written, and should not automatically trigger an event for all existing blocks. However, depending on your existing configuration, the HDFS Balancer will utilize the current rack awareness configuration, such that when it (the HDFS Balancer) is run, it will read the existing blocks and rebalance them based on the current rack awareness topology.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.