Support Questions
Find answers, ask questions, and share your expertise

hadoop cluster + under replica returned again after fixing the previos under replica problem

hadoop cluster + under replica returned again after fixing the previos under replica problem

We have Hadoop cluster with 3 datanodes

 

when we run the test – `Hadoop fsck /` , we found `38763` files with replicated problem

 

to resolves and fix under replica we used the following

 

su - <$hdfs_user>


bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files


bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done

 

The procedure fixed the under replica and it was zero under replica for few hours

 

But after 3 hours under replica return again and by using `Hadoop fsck /`


We can see many of this messages

 

Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s)


What could be the reason that under replica , return again while we resolved the first batch of under replica

 

Michael-Bronson