question Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs. in Archives of Support Questions (Read Only)

NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Mon, 30 Jan 2017 10:27:09 GMT

Hi,

When i Run fsck on my cluster i got that several blocks under replicated and the target replication is 3 even i changed the dfs.replication to NN and DNs to 2.

My cluster status

Live Nodes

3 (Decommissioned: 1)

Total size: 1873902607439 B
Total dirs: 122633
Total files: 117412
Total blocks (validated): 119731 (avg. block size 15650939 B)
Minimally replicated blocks: 119731 (100.0 %)
Over-replicated blocks: 68713 (57.38948 %)
Under-replicated blocks: 27 (0.022550551 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.5738947
Corrupt blocks: 0
Missing replicas: 27 (0.011274004 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Mon Jan 30 04:59:23 EST 2017 in 2468 milliseconds

NN and DNs hdfs.site.xml:

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

The only change i did that i deco one of the servers and it's now in decomissioned state, even i set replication factor for all HDFS manually to 2 but still see the new written blocks are alerted on target replica as 3, also i ensure that the mapred submit replica also 2 in JT:

<property>
<name>mapred.submit.replication</name>
<value>2</value>
</property>

Any insights?

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Tue, 31 Jan 2017 02:26:15 GMT

Anyone who can help here

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mathieu.d — Tue, 31 Jan 2017 09:14:52 GMT

The property you have set is the default replication factor for new blocks. This do not alter existing blocks.

So if you have under-replicated blocks asking for 3 replica then you need to alter these blocks directly.
Check the command line "hdfs dfs -setrep"

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Tue, 31 Jan 2017 19:23:53 GMT

I will ready did, but new written blocks still alerting on under replication

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mbigelow — Fri, 03 Feb 2017 21:12:42 GMT

Can you demonstrate? Show an empty directory, then upload a file, then run hdfs dfs -getrep on the file, and post the output here.

Also post the hdfs-site.xml if possible from the client you are one or were you upload from?

It just dawned on me that you stated that the NN and DN have replication set to 2 but if the clients uploading the data still have it set to 3 in their hdfs-site.xml then the file(s) will have a factor of 3.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Sat, 04 Feb 2017 04:10:32 GMT

I got somehow confused, does this means that client can override the NN
replication definition.

In which cases NN decides the replication and on which client is deciding,
I will check all clients which are working with my cluster and check the
hdfs-site.xml.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mbigelow — Sat, 04 Feb 2017 05:33:42 GMT

The replication factor for a file is set on ingest. And simple, unless you mark it as 'final', a client can override any setting (well some are marked as final in the code base, replication factor is not one of them).

In short, the client always defines the replication factor.

As an example, the Terasort application provide with the distribution has the replication factor set to 1. Unless you change it the terasort output is written to HDFS with only one replica.

The Namenode just keeps track of how many replicas there should be and are, etc.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Sat, 04 Feb 2017 05:41:32 GMT

How i know which blocks are in my cluster with 1 replica, i have several
files and folders and running -ls command will be difficult to follow and
got the block replicas.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mbigelow — Sat, 04 Feb 2017 05:45:58 GMT

hdfs dfs -stat %r /path/to/file (and wildcards work).
This just prints out the factor and not the file name.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Sat, 04 Feb 2017 05:53:32 GMT

Can you please provide an example:

I have /data and /code dirs and under /data i have data1 and data2 dir and
under these data1 and 2 dirs i have files, can you please provide the
command that i ca use?

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Mon, 13 Feb 2017 17:55:52 GMT

If the jobs submitted using oozie and all DNs and NN has replication factor, i checked hdfs-site.xml and mapred-site.xml at all the cluster nodes and all has the value 2, which service i should restart after the change?

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mbigelow — Mon, 13 Feb 2017 18:01:14 GMT

Were you checking /etc/hadoop/conf?

Restart oozie and then find the running process directory under /var/run/cloudera-scm-agent/process. Check the hdfs-site.xml under it to ensure it is set to 2 as well.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Mon, 13 Feb 2017 18:24:12 GMT

yes, I'm looking at /etc/hadoop/conf.

I already tired and restarted the oozie with no success.

I'm using hadoop version 2.0.0-cdh4.3.0, tried to check under /var/run/mapred dirs but find only pid file.

Under /var/run this is what i see:

hald
pm-utils
saslauthd
plymouth
setrans
hadoop-yarn
hadoop-mapreduce
nslcd
console
sepermit
faillock
mdadm
lvm
netreport
ConsoleKit
zookeeper
vmtoolsd.pid
vmware
syslogd.pid
portreserve
auditd.pid
sssd.pid
irqbalance.pid
messagebus.pid
dbus
haldaemon.pid
cupsd.pid
cups
acpid.socket
acpid.pid
xinetd.pid
sshd.pid
nscd
logstash-forwarder.pid
autofs.pid
autofs.fifo-net
autofs.fifo-misc
autofs-running
ntpd.pid
mtstrmd.pid
sm-client.pid
sendmail.pid
abrtd.pid
abrt
hadoop-0.20-mapreduce
crond.pid
cron.reboot
atd.pid
puppet
hsflowd.pid
mcollectived.pid
hadoop-hdfs
zabbix
oozie
utmp

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mbigelow — Mon, 13 Feb 2017 18:27:46 GMT

Did you deploy with Cloudera Manager?

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mbigelow — Mon, 13 Feb 2017 18:28:26 GMT

What is under /var/run/oozie?

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Mon, 13 Feb 2017 18:33:02 GMT

nothing else than oozie.pid

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mbigelow — Mon, 13 Feb 2017 18:45:03 GMT

ok, on the server running oozie run 'ps -ef | grep oozie'. Find the oozie.config.dir value and search it for the configuration files. If there is an hdfs-site.xml there, check it for the repl factor.

Looking at my own CDH 5 cluster I see now that oozie is different than other services it is under /run/cloudera-scm-agent. I don't know if yours will be since you still didn't have the agent process directory under /var/run. I also don't have a hdfs-site.xml under my oozie process directory.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

mbigelow — Mon, 13 Feb 2017 18:46:34 GMT

Disregard my mention of the hdfs-site.xml not being under the oozie process directory. It was under yarn-conf sub-directory.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Mon, 13 Feb 2017 18:58:30 GMT

i have the dir /etc/oozie/conf but there is no hdfs-site.xml.

Yes this cluster is CDH4.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Fawze — Mon, 13 Feb 2017 19:08:33 GMT

No, i'm not using CM