under-replicated blocks

[#1]

I have 18% blocks reported as under-replicated . how can I found out which files are the ones that are not being replicated?

hdfs fsck / -files -blocks -locations is not giving me this info or the information is so huge I am getting lost in it.

[#2 · Accepted solution]

How many nodes is this cluster? This data has a replication factor of 5 but only found 3 replicas. In general, this gets autocorrected as long a there are 5 or more live data nodes. But if you have only 3 node cluster, you will obviously not get 5 replicas.

You can correct number of replicas at HDFS level

hdfs dfs -setrep -w 3 <hdfs path>

I think this is the case of Accumulo using rep factor 5 as default for metadata.

@Josh Elser can you confirm?

[#3]

hi Ravi I installed a 5 node cluster and chose 3 nodes as data nodes , and 1 node as master and 1 node as secondary master .

so how should I have installed and anyway I can correct it?

btw the files he is complaining are not even present in the directory

[hdfs@ip-172-31-11-183 yarn]$ hdfs dfs -ls /apps/accumulo/data/tables/\!0/table_info/ Found 4 items -rw-r--r-- 5 accumulo hdfs 693003 2016-05-02 11:20 /apps/accumulo/data/tables/!0/table_info/A0000189.rf -rw-r--r-- 5 accumulo hdfs 776606 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/A000018b.rf -rw-r--r-- 5 accumulo hdfs 776606 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/A000018c.rf -rw-r--r-- 5 accumulo hdfs 84829 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/F000018a.rf [hdfs@ip-172-31-11-183 yarn]$ [hdfs@ip-172-31-11-183 yarn]$

[#4]

also I just noticed that the all the underreplicated files belong to accumulo , so how are the other files being replicated if I have the replication factor of 5?

[#5]

With 5 nodes, 2 master and 3 datanodes is a good setup. But accumulo by default here is asking for table metadata files to be stores as 5 replicas. While it will not effect running of your system, you will see underreplicated blocks.

Take a look at https://issues.apache.org/jira/browse/ACCUMULO-683

This shows how you can switch the replication factor to 3 for table metadata. Making this change with repfactor on hdfs will not have much effect since any new metadata will again ask for replication factor 5.

[#6]

so is this a bug ? why accumlulo is asking for a replication factor of 5 ?

and how to fix it permanently ?

[#7]

can you please help me understand this link ?

https://issues.apache.org/jira/browse/ACCUMULO-683

iam not sure what fix they are recommending , I tried the config command suggested there but it returns with error saying "!METADATA: event not found"

[#8]

Yep, you're exactly right @Ravi Mutyala. Accumulo increases the default replication on these files to 5 to guard against catastrophic failure resulting in data loss. You can either ignore it, or reduce it by hand via the setrep command.

[#9]

hi Ravi

the command you gave earlier fixed the issue

hdfs dfs -setrep -w 3 /

thanks a lot