Created 06-29-2017 02:01 PM
I was facing failed to replace bad datanode
error while appending new data to file and the work around was to set dfs.replication
to less than 3 , so I set it to 1 just to test it. But I still got the
same error. I looked at the hadoop web interface and surprisingly the
replication factor was still 3 . but when I did hdfs dfs -setrep 1 <file_name>
the replication is set to 1 and I could append to file. Why is this happening? Can I not set default replication factor ?
I tried formating namenode still no change.
Here's my hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>
</configuration>
I tried to follow steps from this question, still my replication factor is 3. I am running hadoop in single node cluster.
Created 06-29-2017 10:50 PM
@Saurab Dahal - As in the other answers the change in hdfs-site.xml will only affect the newly created files with the new replication factor.
If you want to change replication factor for already existing files. You can run the hdfs cli commands like below:
hdfs dfs -setrep -w 1 /myfile hdfs dfs -setrep -w 1 /mydir
Hope this helps !
Created 06-29-2017 05:53 PM
That will only set it for newly created directories. Using the HDFS client, set the replication factor for the directory to the new value.
Created 06-29-2017 05:59 PM
@Saurab Dahal The replication factor is already set to 3 for the file you are trying to append data to. Even if the config value is changed, it gets into effect for new files. Please check the replication factor for a file created after the config value change.
Created 06-29-2017 07:54 PM
Hi Saurab, Are u able to see the changed value in the logs , while restarting the namenode. you can check in the namenode startup logs like.
grep -ri dfs.replication *
Created 06-29-2017 10:50 PM
@Saurab Dahal - As in the other answers the change in hdfs-site.xml will only affect the newly created files with the new replication factor.
If you want to change replication factor for already existing files. You can run the hdfs cli commands like below:
hdfs dfs -setrep -w 1 /myfile hdfs dfs -setrep -w 1 /mydir
Hope this helps !