Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Rack Awareness - Would modifying Rack Awareness topology trigger immediate BLOCK movement ?

Solved Go to solution

Rack Awareness - Would modifying Rack Awareness topology trigger immediate BLOCK movement ?

New Contributor

We have a Cluster running CDH 5.15.1 on 150+ nodes. Cluster nodes storage capacity varies (Mix of various node sizes, 10 TB, 13 TB, 25 TB, 48 TB). 

Nodes are connected with a 10G network and presently running with a physical topology of Rack awareness. 

Issue is, while few racks have 18 servers others are just having 1 or 2 servers on them. we have been advised to move away from this nonuniform rack awareness topology to a Logical Rack awareness topology which will help us put together UNIFORM no. of nodes on each rack.

 

I would like to understand, if the rack awareness topology is changed, would HDFS kick-off immediate Block movement or the new topology will only be effective for NEW blocks written on the cluster. 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Rack Awareness - Would modifying Rack Awareness topology trigger immediate BLOCK movement ?

Contributor

Hi VikramD,

 

Thanks for reaching out.  Regarding your question:

> If the rack awareness topology is changed, would HDFS kick-off immediate Block movement

> or the new topology will only be effective for NEW blocks written on the cluster?

The new topology will be applied when new blocks are written, and should not automatically trigger an event for all existing blocks.   However, depending on your existing configuration, the HDFS Balancer will utilize the current rack awareness configuration, such that when it (the HDFS Balancer) is run, it will read the existing blocks and rebalance them based on the current rack awareness topology.

 

2 REPLIES 2
Highlighted

Re: Rack Awareness - Would modifying Rack Awareness topology trigger immediate BLOCK movement ?

Contributor

Hi VikramD,

 

Thanks for reaching out.  Regarding your question:

> If the rack awareness topology is changed, would HDFS kick-off immediate Block movement

> or the new topology will only be effective for NEW blocks written on the cluster?

The new topology will be applied when new blocks are written, and should not automatically trigger an event for all existing blocks.   However, depending on your existing configuration, the HDFS Balancer will utilize the current rack awareness configuration, such that when it (the HDFS Balancer) is run, it will read the existing blocks and rebalance them based on the current rack awareness topology.

 

Re: Rack Awareness - Would modifying Rack Awareness topology trigger immediate BLOCK movement ?

New Contributor

Thanks much for the quick response.

Don't have an account?
Coming from Hortonworks? Activate your account here