Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

under-replicated blocks

avatar
Super Collaborator

I have 18% blocks reported as under-replicated . how can I found out which files are the ones that are not being replicated?

hdfs fsck / -files -blocks -locations is not giving me this info or the information is so huge I am getting lost in it.

1 ACCEPTED SOLUTION

avatar
Guru

How many nodes is this cluster? This data has a replication factor of 5 but only found 3 replicas. In general, this gets autocorrected as long a there are 5 or more live data nodes. But if you have only 3 node cluster, you will obviously not get 5 replicas.

You can correct number of replicas at HDFS level

hdfs dfs -setrep -w 3 <hdfs path>

I think this is the case of Accumulo using rep factor 5 as default for metadata.

@Josh Elser can you confirm?

View solution in original post

8 REPLIES 8

avatar
Guru

How many nodes is this cluster? This data has a replication factor of 5 but only found 3 replicas. In general, this gets autocorrected as long a there are 5 or more live data nodes. But if you have only 3 node cluster, you will obviously not get 5 replicas.

You can correct number of replicas at HDFS level

hdfs dfs -setrep -w 3 <hdfs path>

I think this is the case of Accumulo using rep factor 5 as default for metadata.

@Josh Elser can you confirm?

avatar
Super Collaborator

hi Ravi I installed a 5 node cluster and chose 3 nodes as data nodes , and 1 node as master and 1 node as secondary master .

so how should I have installed and anyway I can correct it?

btw the files he is complaining are not even present in the directory

[hdfs@ip-172-31-11-183 yarn]$ hdfs dfs -ls /apps/accumulo/data/tables/\!0/table_info/ Found 4 items -rw-r--r-- 5 accumulo hdfs 693003 2016-05-02 11:20 /apps/accumulo/data/tables/!0/table_info/A0000189.rf -rw-r--r-- 5 accumulo hdfs 776606 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/A000018b.rf -rw-r--r-- 5 accumulo hdfs 776606 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/A000018c.rf -rw-r--r-- 5 accumulo hdfs 84829 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/F000018a.rf [hdfs@ip-172-31-11-183 yarn]$ [hdfs@ip-172-31-11-183 yarn]$

avatar
Super Collaborator

also I just noticed that the all the underreplicated files belong to accumulo , so how are the other files being replicated if I have the replication factor of 5?

avatar
Guru

With 5 nodes, 2 master and 3 datanodes is a good setup. But accumulo by default here is asking for table metadata files to be stores as 5 replicas. While it will not effect running of your system, you will see underreplicated blocks.

Take a look at https://issues.apache.org/jira/browse/ACCUMULO-683

This shows how you can switch the replication factor to 3 for table metadata. Making this change with repfactor on hdfs will not have much effect since any new metadata will again ask for replication factor 5.

avatar
Super Collaborator

so is this a bug ? why accumlulo is asking for a replication factor of 5 ?

and how to fix it permanently ?

avatar
Super Collaborator

can you please help me understand this link ?

https://issues.apache.org/jira/browse/ACCUMULO-683

iam not sure what fix they are recommending , I tried the config command suggested there but it returns with error saying "!METADATA: event not found"

avatar
Super Guru

Yep, you're exactly right @Ravi Mutyala. Accumulo increases the default replication on these files to 5 to guard against catastrophic failure resulting in data loss. You can either ignore it, or reduce it by hand via the setrep command.

avatar
Super Collaborator

hi Ravi

the command you gave earlier fixed the issue

hdfs dfs -setrep -w 3 /

thanks a lot