Created 10-21-2016 07:58 AM
Hi,
I had 1 millions of files under replicated in my HDFS cluster for a testing scenario, It replicated successfully but I want to know that Which nodes are participating on this operation ?
Like What is the role of namenode and datanode in this under-replication process ?
Created on 10-21-2016 01:21 PM - edited 08-19-2019 01:08 AM
The ALL are; especially when talking about so many files that were under-replicated. Ultimately, the NN is the one who determines if a file is under-replicated. It is then the NN's job to notify one of the DNs that has a good copy of one of the blocks' replicas to copy it to another DN. NN isn't going to do any of the actual movement of bits -- it will just coordinate the whole effort. Hope this helps!
Created on 10-21-2016 01:21 PM - edited 08-19-2019 01:08 AM
The ALL are; especially when talking about so many files that were under-replicated. Ultimately, the NN is the one who determines if a file is under-replicated. It is then the NN's job to notify one of the DNs that has a good copy of one of the blocks' replicas to copy it to another DN. NN isn't going to do any of the actual movement of bits -- it will just coordinate the whole effort. Hope this helps!
Created 10-21-2016 05:26 PM
More details can be viewed from the "source" at http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html