question Re: HDFS - Under-Replicated Blocks, Corrupt Blocks in Support Questions

HDFS - Under-Replicated Blocks, Corrupt Blocks

NSU — Fri, 16 Sep 2022 09:04:59 GMT

Hi,

I am getting below errors when I ran "hadoop fsck /" command. Please help me on this.

/user/accumulo/.Trash/Current/

accumulo/tables/+r/root_tablet/delete+A00008z4.rf+F00008qz.rf: Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758930_18158. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008z6.rf+A00008z4.rf: Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758933_18161. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008z6.rf+F00008z5.rf: Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758938_18166. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008z8.rf+A00008z6.rf: Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758939_18167. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008z8.rf+F00008z7.rf: Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758941_18169. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008za.rf+A00008z8.rf: Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758942_18170. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008za.rf+F00008z9.rf: Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758944_18172. Target Replicas is 5 but found 3 replica(s).
............................
....................................................................................................
....................................................................................................
......................................................Status: HEALTHY
Total size:    212515269 B (Total open files size: 558 B)
Total dirs:    4197
Total files:    1654
Total symlinks:        0 (Files currently being written: 6)
Total blocks (validated):    1650 (avg. block size 128797 B) (Total open file blocks (not validated): 6)
Minimally replicated blocks:    1650 (100.0 %)
Over-replicated blocks:    0 (0.0 %)
Under-replicated blocks:    1341 (81.27273 %)
Mis-replicated blocks:        0 (0.0 %)
Default replication factor:    3
Average block replication:    2.9921212
Corrupt blocks:        0
Missing replicas:        2669 (35.090717 %)
Number of data-nodes:        3
Number of racks:        1
FSCK ended at Wed Aug 13 19:15:42 EDT 2014 in 77 milliseconds

Re: HDFS - Under-Replicated Blocks, Corrupt Blocks

GautamG — Thu, 14 Aug 2014 00:04:41 GMT

The error message says some Accumulo files in the Trash folder only have 3 replicas whereas there should be 5. The default value of dfs.replication is 3. By default, dfs.replication.max is set to 512. This is the maximum number of replicas for a block. Accumulo checks if dfs.replication.max is set and if not, uses 5 for a replication factor. What version of CDH are you running?

All this is detailed in bug ACCUMULO-683

https://issues.apache.org/jira/browse/ACCUMULO-683

So you can do the following :

- set dfs.replication.max to 3

- set table.file.replication for the !METADATA also to 3

- Use "hadoop fs -setrep" to change the replication factor of those files to 3

http://hadoop.apache.org/docs/r0.18.3/hdfs_shell.html#setrep

- run fsck and confirm you don't get this warning any more

Regards
Gautam

Re: HDFS - Under-Replicated Blocks, Corrupt Blocks

NSU — Thu, 14 Aug 2014 01:09:42 GMT

Thank you for your response. I am using CDH 5

Re: HDFS - Under-Replicated Blocks, Corrupt Blocks

GautamG — Thu, 14 Aug 2014 01:18:56 GMT

https://github.com/cloudera/hadoop-common/blob/cdh5-2.3.0_5.0.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L283

That shows the default in CDH 5.0.0 is 512 as well. Please try the steps I
provided earlier and let me know if it helped.

Re: HDFS - Under-Replicated Blocks, Corrupt Blocks

NSU — Thu, 14 Aug 2014 17:56:20 GMT

Thank you so much.It worked for me.It's a good solution.