Reply
Expert Contributor
Posts: 362
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

If the jobs submitted using oozie and all DNs and NN has replication factor, i checked hdfs-site.xml and mapred-site.xml at all the cluster nodes and all has the value 2, which service i should restart after the change?

 

Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Were you checking /etc/hadoop/conf?

Restart oozie and then find the running process directory under /var/run/cloudera-scm-agent/process. Check the hdfs-site.xml under it to ensure it is set to 2 as well.
Expert Contributor
Posts: 362
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

yes, I'm looking at /etc/hadoop/conf.

 

I already tired and restarted the oozie with no success.

 

I'm using hadoop version 2.0.0-cdh4.3.0, tried to check under /var/run/mapred dirs but find only pid file.

 

Under /var/run this is what i see:

 

hald
pm-utils
saslauthd
plymouth
setrans
hadoop-yarn
hadoop-mapreduce
nslcd
console
sepermit
faillock
mdadm
lvm
netreport
ConsoleKit
zookeeper
vmtoolsd.pid
vmware
syslogd.pid
portreserve
auditd.pid
sssd.pid
irqbalance.pid
messagebus.pid
dbus
haldaemon.pid
cupsd.pid
cups
acpid.socket
acpid.pid
xinetd.pid
sshd.pid
nscd
logstash-forwarder.pid
autofs.pid
autofs.fifo-net
autofs.fifo-misc
autofs-running
ntpd.pid
mtstrmd.pid
sm-client.pid
sendmail.pid
abrtd.pid
abrt
hadoop-0.20-mapreduce
crond.pid
cron.reboot
atd.pid
puppet
hsflowd.pid
mcollectived.pid
hadoop-hdfs
zabbix
oozie
utmp

Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Did you deploy with Cloudera Manager?
Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

What is under /var/run/oozie?
Expert Contributor
Posts: 362
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

nothing else than oozie.pid

Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

ok, on the server running oozie run 'ps -ef | grep oozie'. Find the oozie.config.dir value and search it for the configuration files. If there is an hdfs-site.xml there, check it for the repl factor.

Looking at my own CDH 5 cluster I see now that oozie is different than other services it is under /run/cloudera-scm-agent. I don't know if yours will be since you still didn't have the agent process directory under /var/run. I also don't have a hdfs-site.xml under my oozie process directory.
Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Disregard my mention of the hdfs-site.xml not being under the oozie process directory. It was under yarn-conf sub-directory.
Expert Contributor
Posts: 362
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

i have the dir /etc/oozie/conf but there is no hdfs-site.xml.

 

Yes this cluster is CDH4.

Expert Contributor
Posts: 362
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

No, i'm not using CM