Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Replication factor not changed in HDFS from hdfs-site.xml

avatar
Contributor

I was facing failed to replace bad datanode error while appending new data to file and the work around was to set dfs.replication to less than 3 , so I set it to 1 just to test it. But I still got the same error. I looked at the hadoop web interface and surprisingly the replication factor was still 3 . but when I did hdfs dfs -setrep 1 <file_name> the replication is set to 1 and I could append to file. Why is this happening? Can I not set default replication factor ? I tried formating namenode still no change.

Here's my hdfs-site.xml

<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>

<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

  <property> 
    <name>dfs.support.append</name>
    <value>true</value> 
</property>
</configuration>

I tried to follow steps from this question, still my replication factor is 3. I am running hadoop in single node cluster.

1 ACCEPTED SOLUTION

avatar

@Saurab Dahal - As in the other answers the change in hdfs-site.xml will only affect the newly created files with the new replication factor.

If you want to change replication factor for already existing files. You can run the hdfs cli commands like below:

hdfs dfs -setrep -w 1 /myfile
hdfs dfs -setrep -w 1 /mydir

Hope this helps !

View solution in original post

4 REPLIES 4

avatar

That will only set it for newly created directories. Using the HDFS client, set the replication factor for the directory to the new value.

avatar
Rising Star

@Saurab Dahal The replication factor is already set to 3 for the file you are trying to append data to. Even if the config value is changed, it gets into effect for new files. Please check the replication factor for a file created after the config value change.

avatar
Contributor

Hi Saurab, Are u able to see the changed value in the logs , while restarting the namenode. you can check in the namenode startup logs like.

grep -ri dfs.replication *

avatar

@Saurab Dahal - As in the other answers the change in hdfs-site.xml will only affect the newly created files with the new replication factor.

If you want to change replication factor for already existing files. You can run the hdfs cli commands like below:

hdfs dfs -setrep -w 1 /myfile
hdfs dfs -setrep -w 1 /mydir

Hope this helps !