Hi @hdpadmin overlandpark Thanks for working with me over the Support portal. I will put the steps here for others reference: The Topology information is CACHED in NN and so, when ever there is change to Topology we need to restart the NN to clear it.
In our case Datanode is originally registered without rack (default-rack) and due to caching, NN will not allow it to join when its online.
So to fix this issue, we have to restart namenode, which will pick up the current updated rack information.
To avoid a NN restart, there is a workaround to add a new DataNode to a specific Rack without the need of restarting the NameNode: a. Add the New Node in Ambari without choosing 'DataNode' component
b. From Ambari Hosts tab, Select the new Node and add 'DataNode' component
c. Click 'Host Actions' -> 'Set Rack' and specify the required Rack Name
d. From the Ambari Hosts tab, go to the NameNode Host and Refresh the client configs by dropping down the Menu next to Clients.
This step will update /etc/hadoop/conf/topology_mappings.data on the NameNode with the new topology information for the DN.
e. Start the DataNode service on the new Node.
f. Once the Datanode is up, confirm its Rack topology information, by running the following command on any node with hdfs client:
# su - hdfs
# hdfs dfsadmin -report
In short, the Topology should be updated before a Datanode is started (at which point it tries to register with the NN)
... View more
