Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

avatar
Master Collaborator

Hi,

 

When i Run fsck on my cluster i got that several blocks under replicated and the target replication is 3 even i changed the dfs.replication to NN and DNs to 2.

 

My cluster status

 

Live Nodes:3 (Decommissioned: 1)

 

 

Total size: 1873902607439 B
Total dirs: 122633
Total files: 117412
Total blocks (validated): 119731 (avg. block size 15650939 B)
Minimally replicated blocks: 119731 (100.0 %)
Over-replicated blocks: 68713 (57.38948 %)
Under-replicated blocks: 27 (0.022550551 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.5738947
Corrupt blocks: 0
Missing replicas: 27 (0.011274004 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Jan 30 04:59:23 EST 2017 in 2468 milliseconds

 

 

NN and DNs hdfs.site.xml:

 

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

 

The only change i did that i deco one of the servers and it's now in decomissioned state, even i set replication factor for all HDFS manually to 2 but still see the new written blocks are alerted on target replica as 3, also i ensure that the mapred submit replica also 2 in JT:

 

<property>
<name>mapred.submit.replication</name>
<value>2</value>
</property>

 

Any insights?

 

 

1 ACCEPTED SOLUTION

avatar
Champion
Lets step back, instead of trying to hunt down were it is set on the client side mark dfs.replication to final in your configs. This will prevent any clients from changing it at run time.

<property>
<name>dfs.replication</name>
<value>2</value>
<final>true</final>
</property>

View solution in original post

24 REPLIES 24

avatar
Master Collaborator

Anyone who can help here

avatar
Super Collaborator
The property you have set is the default replication factor for new blocks. This do not alter existing blocks.

So if you have under-replicated blocks asking for 3 replica then you need to alter these blocks directly.
Check the command line "hdfs dfs -setrep"

avatar
Master Collaborator
I will ready did, but new written blocks still alerting on under replication

avatar
Champion
Can you demonstrate? Show an empty directory, then upload a file, then run hdfs dfs -getrep on the file, and post the output here.

Also post the hdfs-site.xml if possible from the client you are one or were you upload from?

It just dawned on me that you stated that the NN and DN have replication set to 2 but if the clients uploading the data still have it set to 3 in their hdfs-site.xml then the file(s) will have a factor of 3.

avatar
Master Collaborator
I got somehow confused, does this means that client can override the NN
replication definition.

In which cases NN decides the replication and on which client is deciding,
I will check all clients which are working with my cluster and check the
hdfs-site.xml.

avatar
Champion
The replication factor for a file is set on ingest. And simple, unless you mark it as 'final', a client can override any setting (well some are marked as final in the code base, replication factor is not one of them).

In short, the client always defines the replication factor.

As an example, the Terasort application provide with the distribution has the replication factor set to 1. Unless you change it the terasort output is written to HDFS with only one replica.

The Namenode just keeps track of how many replicas there should be and are, etc.

avatar
Master Collaborator
How i know which blocks are in my cluster with 1 replica, i have several
files and folders and running -ls command will be difficult to follow and
got the block replicas.

avatar
Champion
hdfs dfs -stat %r /path/to/file (and wildcards work).
This just prints out the factor and not the file name.

avatar
Master Collaborator
Can you please provide an example:

I have /data and /code dirs and under /data i have data1 and data2 dir and
under these data1 and 2 dirs i have files, can you please provide the
command that i ca use?