NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

[#1]

Hi,

When i Run fsck on my cluster i got that several blocks under replicated and the target replication is 3 even i changed the dfs.replication to NN and DNs to 2.

My cluster status

Live Nodes

:

3 (Decommissioned: 1)

Total size: 1873902607439 B
Total dirs: 122633
Total files: 117412
Total blocks (validated): 119731 (avg. block size 15650939 B)
Minimally replicated blocks: 119731 (100.0 %)
Over-replicated blocks: 68713 (57.38948 %)
Under-replicated blocks: 27 (0.022550551 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.5738947
Corrupt blocks: 0
Missing replicas: 27 (0.011274004 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Jan 30 04:59:23 EST 2017 in 2468 milliseconds

NN and DNs hdfs.site.xml:

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

The only change i did that i deco one of the servers and it's now in decomissioned state, even i set replication factor for all HDFS manually to 2 but still see the new written blocks are alerted on target replica as 3, also i ensure that the mapred submit replica also 2 in JT:

<property>
<name>mapred.submit.replication</name>
<value>2</value>
</property>

Any insights?

[#2]

Anyone who can help here

[#3]

The property you have set is the default replication factor for new blocks. This do not alter existing blocks.

So if you have under-replicated blocks asking for 3 replica then you need to alter these blocks directly.
Check the command line "hdfs dfs -setrep"

[#4]

I will ready did, but new written blocks still alerting on under replication

[#5]

Can you demonstrate? Show an empty directory, then upload a file, then run hdfs dfs -getrep on the file, and post the output here.

Also post the hdfs-site.xml if possible from the client you are one or were you upload from?

It just dawned on me that you stated that the NN and DN have replication set to 2 but if the clients uploading the data still have it set to 3 in their hdfs-site.xml then the file(s) will have a factor of 3.

[#6]

I got somehow confused, does this means that client can override the NN
replication definition.

In which cases NN decides the replication and on which client is deciding,
I will check all clients which are working with my cluster and check the
hdfs-site.xml.

[#7]

The replication factor for a file is set on ingest. And simple, unless you mark it as 'final', a client can override any setting (well some are marked as final in the code base, replication factor is not one of them).

In short, the client always defines the replication factor.

As an example, the Terasort application provide with the distribution has the replication factor set to 1. Unless you change it the terasort output is written to HDFS with only one replica.

The Namenode just keeps track of how many replicas there should be and are, etc.

[#8]

How i know which blocks are in my cluster with 1 replica, i have several
files and folders and running -ls command will be difficult to follow and
got the block replicas.

[#9]

hdfs dfs -stat %r /path/to/file (and wildcards work).
This just prints out the factor and not the file name.

[#10]

Can you please provide an example:

I have /data and /code dirs and under /data i have data1 and data2 dir and
under these data1 and 2 dirs i have files, can you please provide the
command that i ca use?