Support Questions

Find answers, ask questions, and share your expertise

Rack Awareness - Would modifying Rack Awareness topology trigger immediate BLOCK movement ?

avatar
New Contributor

We have a Cluster running CDH 5.15.1 on 150+ nodes. Cluster nodes storage capacity varies (Mix of various node sizes, 10 TB, 13 TB, 25 TB, 48 TB). 

Nodes are connected with a 10G network and presently running with a physical topology of Rack awareness. 

Issue is, while few racks have 18 servers others are just having 1 or 2 servers on them. we have been advised to move away from this nonuniform rack awareness topology to a Logical Rack awareness topology which will help us put together UNIFORM no. of nodes on each rack.

 

I would like to understand, if the rack awareness topology is changed, would HDFS kick-off immediate Block movement or the new topology will only be effective for NEW blocks written on the cluster. 

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Hi VikramD,

 

Thanks for reaching out.  Regarding your question:

> If the rack awareness topology is changed, would HDFS kick-off immediate Block movement

> or the new topology will only be effective for NEW blocks written on the cluster?

The new topology will be applied when new blocks are written, and should not automatically trigger an event for all existing blocks.   However, depending on your existing configuration, the HDFS Balancer will utilize the current rack awareness configuration, such that when it (the HDFS Balancer) is run, it will read the existing blocks and rebalance them based on the current rack awareness topology.

 

View solution in original post

2 REPLIES 2

avatar
Expert Contributor

Hi VikramD,

 

Thanks for reaching out.  Regarding your question:

> If the rack awareness topology is changed, would HDFS kick-off immediate Block movement

> or the new topology will only be effective for NEW blocks written on the cluster?

The new topology will be applied when new blocks are written, and should not automatically trigger an event for all existing blocks.   However, depending on your existing configuration, the HDFS Balancer will utilize the current rack awareness configuration, such that when it (the HDFS Balancer) is run, it will read the existing blocks and rebalance them based on the current rack awareness topology.

 

avatar
New Contributor

Thanks much for the quick response.