Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Replication factor in HDFS

Highlighted

Replication factor in HDFS

New Contributor

Hi,

I have setup a 2 nodes HDFS cluster and given replication factor 2. Now while I am trying to upload a new file it is replicating the files block in both data nodes but it still consider the 3rd replication as a under replicated blocks.How to resolve this ?

I need only 2 exact copy of file i.e dfs.replication = 2.

7 REPLIES 7

Re: Replication factor in HDFS

@Viraj Vekaria

Can you please check and make sure the property has taken effect ? Seems the property has not taken in affect.

You have test using setting replication using cli - $hdfs dfs -setrep 2 <filename>

Re: Replication factor in HDFS

New Contributor

I don't want to set every time I am uploading the file, instead I have changed in the hdfs-site.xml file to 2. This should work if I am not wrong.

I am copying the whole directory from local to HDFS, How am I supposed to set the replication factor to 2 each time file is being uploaded.

Re: Replication factor in HDFS

@Viraj Vekaria Its for sake of testing as per @Ashnee Sharma mentioned. Also make sure if all the nodes have hdfs-site.xml reflected with replication factor value 2.

Re: Replication factor in HDFS

You'll have to re-apply the replication factor the the directories you're seeing this warning on. The dfs.replication setting is applied to directories/files at time of creation. And if the cluster was initially created when the value was set to 3 (default), then all the clusters files and folders created at that time will have this applied already. You'll need to reset it for those directories. And new replication factor will be picked up for new files.

Re: Replication factor in HDFS

New Contributor

I got your point, but if I am changing this configuration dfs.replication in ambari to 2, than it should consider right ?

concern is why this is not considering it yet I have changed the default value of dfs.replication to 2. and how to re-apply the replication factor as you mentioned in above comment , I am copying the whole directory contains 50GB of documents.

Re: Replication factor in HDFS

@ Viraj Vekaria

In previous comment he just asked you to set for ontime, and see if property is working from command line, then the issue with your configuration which is done by you in hdfs.hite.xml. It's mean property is not set properly or not reflected.

Re: Replication factor in HDFS

New Contributor

Make sure you have the property "dfs.replication" propagated to hdfs-site.xml on all the below. NN, DN1, DN2, DN3, etc....

Don't have an account?
Coming from Hortonworks? Activate your account here