Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

SOLVED Go to solution

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Super Collaborator

If the jobs submitted using oozie and all DNs and NN has replication factor, i checked hdfs-site.xml and mapred-site.xml at all the cluster nodes and all has the value 2, which service i should restart after the change?

 

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Champion
Were you checking /etc/hadoop/conf?

Restart oozie and then find the running process directory under /var/run/cloudera-scm-agent/process. Check the hdfs-site.xml under it to ensure it is set to 2 as well.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Super Collaborator

yes, I'm looking at /etc/hadoop/conf.

 

I already tired and restarted the oozie with no success.

 

I'm using hadoop version 2.0.0-cdh4.3.0, tried to check under /var/run/mapred dirs but find only pid file.

 

Under /var/run this is what i see:

 

hald
pm-utils
saslauthd
plymouth
setrans
hadoop-yarn
hadoop-mapreduce
nslcd
console
sepermit
faillock
mdadm
lvm
netreport
ConsoleKit
zookeeper
vmtoolsd.pid
vmware
syslogd.pid
portreserve
auditd.pid
sssd.pid
irqbalance.pid
messagebus.pid
dbus
haldaemon.pid
cupsd.pid
cups
acpid.socket
acpid.pid
xinetd.pid
sshd.pid
nscd
logstash-forwarder.pid
autofs.pid
autofs.fifo-net
autofs.fifo-misc
autofs-running
ntpd.pid
mtstrmd.pid
sm-client.pid
sendmail.pid
abrtd.pid
abrt
hadoop-0.20-mapreduce
crond.pid
cron.reboot
atd.pid
puppet
hsflowd.pid
mcollectived.pid
hadoop-hdfs
zabbix
oozie
utmp

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Champion
Did you deploy with Cloudera Manager?

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Super Collaborator

No, i'm not using CM

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Champion
Lets step back, instead of trying to hunt down were it is set on the client side mark dfs.replication to final in your configs. This will prevent any clients from changing it at run time.

<property>
<name>dfs.replication</name>
<value>2</value>
<final>true</final>
</property>

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Super Collaborator

Changed at all the cluster nodes and restarted all services at the cluster after.

 

It didn't solve the issue.

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Super Collaborator

Looking at one of  the running jobs conf and see the following with replication factor 3:

 

mapreduce.client.submit.file.replication

s3.replication

kfs.replication

dfs.namenode.replication.interval

ftp.replication

s3native.replication

 

 

 

 

Highlighted

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Super Collaborator

Any other ideas?

 

The more intersting in the issue that it's happens only for the output of specific jobs and notf or all the HDFS.

 

Is there any way to set that the new written files to specific dir to be with specific replication factor?

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Super Collaborator

Digging down in the cluster, i found one of the application that runs outside of the hadoop cluster has clients that make hdfs dfs -put to the hadoop cluster, these clients weren't have hdfs-site.xml and it got the default replication factor for the cluster, what i did? tested the hdfs dfs -put from a cleint server in my cluster and the client out side the cluster and notice the client outside the cluster put files with replication factor 3, to solve the issue i added hdfs-site.xml to each of the clients outside the cluster and override the default replication factor at the file.