Created 07-19-2017 07:42 AM
We are using Hortonworks HDP 2.1 (HDFS 2.4), with replication factor 3. We have recently decommissioned a datanode and that left a lot of under replicated blocks in the cluster.
Cluster is now trying to satisfy the replication factor by distributing under replicated blocks among other nodes.
Appreciate your help on this
Created 07-19-2017 07:52 AM
1 . First you need to run hadoop fsck / to check the under-replicated blocks. Then you can run hadoop -setrep 2 to the files which are under-replicated. This will stop the process.
2. Yes it will remove the third copy.
Created 07-19-2017 09:24 AM
So eventually, -setrep 2 stops the process and spawn another process (deleting the third copy) right? Is there any way to stop the cluster removing the third copy? I'm trying to reduce the cpu utilization.
Created 07-20-2017 09:32 AM
No, Namenode does this autometically
Created 07-19-2017 09:26 AM
Would you be able to share any references on the second answer?
Created 07-20-2017 09:32 AM
Have a look at this http://www.aosabook.org/en/hdfs.html