question Re: under-replicated blocks in Archives of Support Questions (Read Only)

under-replicated blocks

aliyesami — Sat, 30 Apr 2016 04:53:52 GMT

I have 18% blocks reported as under-replicated . how can I found out which files are the ones that are not being replicated?

hdfs fsck / -files -blocks -locations is not giving me this info or the information is so huge I am getting lost in it.

Re: under-replicated blocks

ravi1 — Mon, 02 May 2016 22:20:35 GMT

How many nodes is this cluster? This data has a replication factor of 5 but only found 3 replicas. In general, this gets autocorrected as long a there are 5 or more live data nodes. But if you have only 3 node cluster, you will obviously not get 5 replicas.

You can correct number of replicas at HDFS level

hdfs dfs -setrep -w 3 <hdfs path>

I think this is the case of Accumulo using rep factor 5 as default for metadata.

@Josh Elser can you confirm?

Re: under-replicated blocks

aliyesami — Mon, 02 May 2016 22:28:54 GMT

hi Ravi I installed a 5 node cluster and chose 3 nodes as data nodes , and 1 node as master and 1 node as secondary master .

so how should I have installed and anyway I can correct it?

btw the files he is complaining are not even present in the directory

[hdfs@ip-172-31-11-183 yarn]$ hdfs dfs -ls /apps/accumulo/data/tables/\!0/table_info/ Found 4 items -rw-r--r-- 5 accumulo hdfs 693003 2016-05-02 11:20 /apps/accumulo/data/tables/!0/table_info/A0000189.rf -rw-r--r-- 5 accumulo hdfs 776606 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/A000018b.rf -rw-r--r-- 5 accumulo hdfs 776606 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/A000018c.rf -rw-r--r-- 5 accumulo hdfs 84829 2016-05-02 11:25 /apps/accumulo/data/tables/!0/table_info/F000018a.rf [hdfs@ip-172-31-11-183 yarn]$ [hdfs@ip-172-31-11-183 yarn]$

Re: under-replicated blocks

aliyesami — Mon, 02 May 2016 22:49:58 GMT

also I just noticed that the all the underreplicated files belong to accumulo , so how are the other files being replicated if I have the replication factor of 5?

Re: under-replicated blocks

ravi1 — Mon, 02 May 2016 23:39:17 GMT

With 5 nodes, 2 master and 3 datanodes is a good setup. But accumulo by default here is asking for table metadata files to be stores as 5 replicas. While it will not effect running of your system, you will see underreplicated blocks.

Take a look at https://issues.apache.org/jira/browse/ACCUMULO-683

This shows how you can switch the replication factor to 3 for table metadata. Making this change with repfactor on hdfs will not have much effect since any new metadata will again ask for replication factor 5.

Re: under-replicated blocks

aliyesami — Mon, 02 May 2016 23:51:25 GMT

so is this a bug ? why accumlulo is asking for a replication factor of 5 ?

and how to fix it permanently ?

Re: under-replicated blocks

aliyesami — Tue, 03 May 2016 00:01:43 GMT

can you please help me understand this link ?

https://issues.apache.org/jira/browse/ACCUMULO-683

iam not sure what fix they are recommending , I tried the config command suggested there but it returns with error saying "!METADATA: event not found"

Re: under-replicated blocks

aliyesami — Tue, 03 May 2016 00:23:32 GMT

hi Ravi

the command you gave earlier fixed the issue

hdfs dfs -setrep -w 3 /

thanks a lot

Re: under-replicated blocks

elserj — Tue, 03 May 2016 00:58:45 GMT

Yep, you're exactly right @Ravi Mutyala. Accumulo increases the default replication on these files to 5 to guard against catastrophic failure resulting in data loss. You can either ignore it, or reduce it by hand via the setrep command.