The hdfs fsck operation doesn't check blocks for corruption; that takes too long. It looks at the directory structures alone. Blocks are checked for corruption whenever they are read; there are little CRC checksum files created for parts of a block which are validated on read() operations. If you work with the file:// filesystem you can see these same files in your local FS. If a block is found to be corrupt on a read, the dfs client will report this to the namenode, and ask for another block, which will be used instead. As Chris said, the namenode then schedules the uncorrupted block for re-replication, as if it was under replicated. The corrupted block doesn't get deleted until that replication succeeds. Why not? If all blocks are corrupt, then maybe you can salvage something from all the corrupt copies of the block. Datanodes scan all files in the background —they just do it fairly slowly by default so that applications don't suffer. The scan ensures that corrupted blocks are usually found before programs read them, and so that problems with "cold" data are found at all. It's designed to avoid the problem of all replicas getting corrupted and you not noticing until its too late to fix. Look in the HDFS XML description for the details on the two options you need to adjust dfs.datanode.scan.period.hours
dfs.block.scanner.volume.bytes.per.second How disks fail/data gets corrupt is a fascinating problem. Here are some links if you really want to learn more about it Did you really want that data (an old presentation of mine) Failure Trends in a Large Disk Drive Population (google) A Large-Scale Study of Flash Memory Failures in the Field (a recent facebook paper on Flash failures -shows they are less common than you'd fear) I'd also recommend you look at some of the work on memory corruption -that's enough to make you think that modern laptops and desktops should be using ECC RAM.
... View more
