Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

avatar
Master Collaborator

Hi,

 

When i Run fsck on my cluster i got that several blocks under replicated and the target replication is 3 even i changed the dfs.replication to NN and DNs to 2.

 

My cluster status

 

Live Nodes:3 (Decommissioned: 1)

 

 

Total size: 1873902607439 B
Total dirs: 122633
Total files: 117412
Total blocks (validated): 119731 (avg. block size 15650939 B)
Minimally replicated blocks: 119731 (100.0 %)
Over-replicated blocks: 68713 (57.38948 %)
Under-replicated blocks: 27 (0.022550551 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.5738947
Corrupt blocks: 0
Missing replicas: 27 (0.011274004 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Jan 30 04:59:23 EST 2017 in 2468 milliseconds

 

 

NN and DNs hdfs.site.xml:

 

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

 

The only change i did that i deco one of the servers and it's now in decomissioned state, even i set replication factor for all HDFS manually to 2 but still see the new written blocks are alerted on target replica as 3, also i ensure that the mapred submit replica also 2 in JT:

 

<property>
<name>mapred.submit.replication</name>
<value>2</value>
</property>

 

Any insights?

 

 

1 ACCEPTED SOLUTION

avatar
Champion
Lets step back, instead of trying to hunt down were it is set on the client side mark dfs.replication to final in your configs. This will prevent any clients from changing it at run time.

<property>
<name>dfs.replication</name>
<value>2</value>
<final>true</final>
</property>

View solution in original post

24 REPLIES 24

avatar
Champion
What is under /var/run/oozie?

avatar
Master Collaborator

nothing else than oozie.pid

avatar
Champion
ok, on the server running oozie run 'ps -ef | grep oozie'. Find the oozie.config.dir value and search it for the configuration files. If there is an hdfs-site.xml there, check it for the repl factor.

Looking at my own CDH 5 cluster I see now that oozie is different than other services it is under /run/cloudera-scm-agent. I don't know if yours will be since you still didn't have the agent process directory under /var/run. I also don't have a hdfs-site.xml under my oozie process directory.

avatar
Master Collaborator

i have the dir /etc/oozie/conf but there is no hdfs-site.xml.

 

Yes this cluster is CDH4.

avatar
Champion
Disregard my mention of the hdfs-site.xml not being under the oozie process directory. It was under yarn-conf sub-directory.