NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

[#1]

Hi,

When i Run fsck on my cluster i got that several blocks under replicated and the target replication is 3 even i changed the dfs.replication to NN and DNs to 2.

My cluster status

Live Nodes

:

3 (Decommissioned: 1)

Total size: 1873902607439 B
Total dirs: 122633
Total files: 117412
Total blocks (validated): 119731 (avg. block size 15650939 B)
Minimally replicated blocks: 119731 (100.0 %)
Over-replicated blocks: 68713 (57.38948 %)
Under-replicated blocks: 27 (0.022550551 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.5738947
Corrupt blocks: 0
Missing replicas: 27 (0.011274004 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Jan 30 04:59:23 EST 2017 in 2468 milliseconds

NN and DNs hdfs.site.xml:

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

The only change i did that i deco one of the servers and it's now in decomissioned state, even i set replication factor for all HDFS manually to 2 but still see the new written blocks are alerted on target replica as 3, also i ensure that the mapred submit replica also 2 in JT:

<property>
<name>mapred.submit.replication</name>
<value>2</value>
</property>

Any insights?

[#2]

What is under /var/run/oozie?

[#3]

nothing else than oozie.pid

[#4]

ok, on the server running oozie run 'ps -ef | grep oozie'. Find the oozie.config.dir value and search it for the configuration files. If there is an hdfs-site.xml there, check it for the repl factor.

Looking at my own CDH 5 cluster I see now that oozie is different than other services it is under /run/cloudera-scm-agent. I don't know if yours will be since you still didn't have the agent process directory under /var/run. I also don't have a hdfs-site.xml under my oozie process directory.

[#5]

i have the dir /etc/oozie/conf but there is no hdfs-site.xml.

Yes this cluster is CDH4.

[#6]

Disregard my mention of the hdfs-site.xml not being under the oozie process directory. It was under yarn-conf sub-directory.