First of all, CDH didn't support SAN until very recently, and even now the support is limited. https://www.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide.html Warning: Running CDH on storage platforms other than direct-attached physical disks can provide suboptimal performance. Cloudera Enterprise and the majority of the Hadoop platform are optimized to provide high performance by distributing work across a cluster that can utilize data locality and fast local I/O. Refer to the Cloudera Enterprise Storage Device Acceptance Criteria Guide for more information about using non-local storage. That said, I am interested in knowing more about your setup. What application writes those corrupt files? The HDFS in CDH5.15 is quite stable and most of the known data corruption bugs were fixed. Probably the only bug not in CDH5.15 is HDFS-10240, where Flume in a busy cluster could trigger this bug. But the symptom doesn't quite match your description anyway. > - Is it possible for -verifyMeta check to fail but actual checksum verification (as part of serving the block to client) to pass on the datanode as we saw? I won't say that's impossible. But we've not seen such a case. The -verifyMeta implementation is quite simple actually. - Should all replicas of the block have same hash (say MD5)? If all replicas have the same size, they are supposed to have the same checksum. (We support append, no truncate.) If your SAN device is busy, there are chances where HDFS client would give up writing to the DataNode, replicate the blocks to a different DataNode and continue from there. In which case, replicas may have different file length, because some of the are stale. - What may be causing finalized blocks to start failing checksum errors if disk is healthy? An under performed disk or a busy DataNode could abort the write to that block. I can't give you a definitive answer because I don't have much experience with HDFS on SAN.
... View more
