question Re: Replication factor not changed in HDFS from hdfs-site.xml in Archives of Support Questions (Read Only)

Replication factor not changed in HDFS from hdfs-site.xml

d_saurab1 — Thu, 29 Jun 2017 21:01:17 GMT

I was facing failed to replace bad datanode error while appending new data to file and the work around was to set dfs.replication to less than 3 , so I set it to 1 just to test it. But I still got the same error. I looked at the hadoop web interface and surprisingly the replication factor was still 3 . but when I did hdfs dfs -setrep 1 <file_name> the replication is set to 1 and I could append to file. Why is this happening? Can I not set default replication factor ? I tried formating namenode still no change.

Here's my hdfs-site.xml

<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>

<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

  <property> 
    <name>dfs.support.append</name>
    <value>true</value> 
</property>
</configuration>

I tried to follow steps from this question, still my replication factor is 3. I am running hadoop in single node cluster.

Re: Replication factor not changed in HDFS from hdfs-site.xml

dstreev — Fri, 30 Jun 2017 00:53:08 GMT

That will only set it for newly created directories. Using the HDFS client, set the replication factor for the directory to the new value.

Re: Replication factor not changed in HDFS from hdfs-site.xml

ssanthosh — Fri, 30 Jun 2017 00:59:35 GMT

@Saurab Dahal The replication factor is already set to 3 for the file you are trying to append data to. Even if the config value is changed, it gets into effect for new files. Please check the replication factor for a file created after the config value change.

Re: Replication factor not changed in HDFS from hdfs-site.xml

kalaiselvan_moo — Fri, 30 Jun 2017 02:54:10 GMT

Hi Saurab, Are u able to see the changed value in the logs , while restarting the namenode. you can check in the namenode startup logs like.

grep -ri dfs.replication *

Re: Replication factor not changed in HDFS from hdfs-site.xml

namaheshwari — Fri, 30 Jun 2017 05:50:12 GMT

@Saurab Dahal - As in the other answers the change in hdfs-site.xml will only affect the newly created files with the new replication factor.

If you want to change replication factor for already existing files. You can run the hdfs cli commands like below:

hdfs dfs -setrep -w 1 /myfile
hdfs dfs -setrep -w 1 /mydir

Hope this helps !