HDFS - Under-Replicated Blocks, Corrupt Blocks

[#1]

Hi,

I am getting below errors when I ran "hadoop fsck /" command. Please help me on this. 

/user/accumulo/.Trash/Current/

accumulo/tables/+r/root_tablet/delete+A00008z4.rf+F00008qz.rf:  Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758930_18158. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008z6.rf+A00008z4.rf:  Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758933_18161. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008z6.rf+F00008z5.rf:  Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758938_18166. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008z8.rf+A00008z6.rf:  Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758939_18167. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008z8.rf+F00008z7.rf:  Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758941_18169. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008za.rf+A00008z8.rf:  Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758942_18170. Target Replicas is 5 but found 3 replica(s).
.
/user/accumulo/.Trash/Current/accumulo/tables/+r/root_tablet/delete+A00008za.rf+F00008z9.rf:  Under replicated BP-349021044-10.180.6.236-1406825419975:blk_1073758944_18172. Target Replicas is 5 but found 3 replica(s).
............................
....................................................................................................
....................................................................................................
......................................................Status: HEALTHY
 Total size:    212515269 B (Total open files size: 558 B)
 Total dirs:    4197
 Total files:    1654
 Total symlinks:        0 (Files currently being written: 6)
 Total blocks (validated):    1650 (avg. block size 128797 B) (Total open file blocks (not validated): 6)
 Minimally replicated blocks:    1650 (100.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:    1341 (81.27273 %)
 Mis-replicated blocks:        0 (0.0 %)
 Default replication factor:    3
 Average block replication:    2.9921212
 Corrupt blocks:        0
 Missing replicas:        2669 (35.090717 %)
 Number of data-nodes:        3
 Number of racks:        1
FSCK ended at Wed Aug 13 19:15:42 EDT 2014 in 77 milliseconds

[#2 · Accepted solution]

The error message says some Accumulo files in the Trash folder only have 3 replicas whereas there should be 5. The default value of dfs.replication is 3. By default, dfs.replication.max is set to 512. This is the maximum number of replicas for a block. Accumulo checks if dfs.replication.max is set and if not, uses 5 for a replication factor. What version of CDH are you running?

All this is detailed in bug ACCUMULO-683

https://issues.apache.org/jira/browse/ACCUMULO-683

 So you can do the following :

- set dfs.replication.max to 3

- set table.file.replication for the !METADATA also to 3

- Use "hadoop fs -setrep" to change the replication factor of those files to 3

http://hadoop.apache.org/docs/r0.18.3/hdfs_shell.html#setrep

- run fsck and confirm you don't get this warning any more

Regards
Gautam

[#3]

Thank you for your response. I am using CDH 5

[#4]

https://github.com/cloudera/hadoop-common/blob/cdh5-2.3.0_5.0.0/hadoop-hdfs-project/hadoop-hdfs/src/...

That shows the default in CDH 5.0.0 is 512 as well. Please try the steps I
provided earlier and let me know if it helped.

[#5]

Thank you so much.It worked for me.It's a good solution.