- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to reduce the replication factor in a HDFS directory and it's impact
- Labels:
-
Apache Hadoop
Created ‎07-19-2017 07:42 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are using Hortonworks HDP 2.1 (HDFS 2.4), with replication factor 3. We have recently decommissioned a datanode and that left a lot of under replicated blocks in the cluster.
Cluster is now trying to satisfy the replication factor by distributing under replicated blocks among other nodes.
- How do I stop that process. I am OK with some files being replicated only twice. If I change the replication factor to 2 in that directory, will that process be terminated?
- What's the impact of making the replication factor to 2 for a directory which has files with 3 copies. Will the cluster start another process to remove the excess copy for each file with 3 copies?
Appreciate your help on this
Created ‎07-19-2017 07:52 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
1 . First you need to run hadoop fsck / to check the under-replicated blocks. Then you can run hadoop -setrep 2 to the files which are under-replicated. This will stop the process.
2. Yes it will remove the third copy.
Created ‎07-19-2017 09:24 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So eventually, -setrep 2 stops the process and spawn another process (deleting the third copy) right? Is there any way to stop the cluster removing the third copy? I'm trying to reduce the cpu utilization.
Created ‎07-20-2017 09:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No, Namenode does this autometically
Created ‎07-19-2017 09:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Would you be able to share any references on the second answer?
Created ‎07-20-2017 09:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Have a look at this http://www.aosabook.org/en/hdfs.html
