Replication factor in HDFS

[#1]

Hi,

I have setup a 2 nodes HDFS cluster and given replication factor 2. Now while I am trying to upload a new file it is replicating the files block in both data nodes but it still consider the 3rd replication as a under replicated blocks.How to resolve this ?

I need only 2 exact copy of file i.e dfs.replication = 2.

[#2]

@Viraj Vekaria

Can you please check and make sure the property has taken effect ? Seems the property has not taken in affect.

You have test using setting replication using cli - $hdfs dfs -setrep 2 <filename>

[#3]

I don't want to set every time I am uploading the file, instead I have changed in the hdfs-site.xml file to 2. This should work if I am not wrong.

I am copying the whole directory from local to HDFS, How am I supposed to set the replication factor to 2 each time file is being uploaded.

[#4]

@Viraj Vekaria Its for sake of testing as per @Ashnee Sharma mentioned. Also make sure if all the nodes have hdfs-site.xml reflected with replication factor value 2.

[#5]

You'll have to re-apply the replication factor the the directories you're seeing this warning on. The dfs.replication setting is applied to directories/files at time of creation. And if the cluster was initially created when the value was set to 3 (default), then all the clusters files and folders created at that time will have this applied already. You'll need to reset it for those directories. And new replication factor will be picked up for new files.

[#6]

I got your point, but if I am changing this configuration dfs.replication in ambari to 2, than it should consider right ?

concern is why this is not considering it yet I have changed the default value of dfs.replication to 2. and how to re-apply the replication factor as you mentioned in above comment , I am copying the whole directory contains 50GB of documents.

[#7]

@ Viraj Vekaria

In previous comment he just asked you to set for ontime, and see if property is working from command line, then the issue with your configuration which is done by you in hdfs.hite.xml. It's mean property is not set properly or not reflected.

[#8]

Make sure you have the property "dfs.replication" propagated to hdfs-site.xml on all the below. NN, DN1, DN2, DN3, etc....