Reply
Posts: 642
Topics: 3
Kudos: 120
Solutions: 67
Registered: ‎08-16-2016

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Lets step back, instead of trying to hunt down were it is set on the client side mark dfs.replication to final in your configs. This will prevent any clients from changing it at run time.

<property>
<name>dfs.replication</name>
<value>2</value>
<final>true</final>
</property>
Expert Contributor
Posts: 357
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Changed at all the cluster nodes and restarted all services at the cluster after.

 

It didn't solve the issue.

Expert Contributor
Posts: 357
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Looking at one of  the running jobs conf and see the following with replication factor 3:

 

mapreduce.client.submit.file.replication

s3.replication

kfs.replication

dfs.namenode.replication.interval

ftp.replication

s3native.replication

 

 

 

 

Expert Contributor
Posts: 357
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

[ Edited ]

Any other ideas?

 

The more intersting in the issue that it's happens only for the output of specific jobs and notf or all the HDFS.

 

Is there any way to set that the new written files to specific dir to be with specific replication factor?

Expert Contributor
Posts: 357
Registered: ‎01-25-2017

Re: NameNode alerting on Blocks under replicated event dfs replication changed at NN and DNs.

Digging down in the cluster, i found one of the application that runs outside of the hadoop cluster has clients that make hdfs dfs -put to the hadoop cluster, these clients weren't have hdfs-site.xml and it got the default replication factor for the cluster, what i did? tested the hdfs dfs -put from a cleint server in my cluster and the client out side the cluster and notice the client outside the cluster put files with replication factor 3, to solve the issue i added hdfs-site.xml to each of the clients outside the cluster and override the default replication factor at the file.

Announcements