The HDFS service in Cloudera Manager is reporting 1 missing block (which is a tiny %) but if I starting digging for the issue using `hdfs fsck` it reports that everything is healthy (ie no corrupt or missing blocks).
Any ideas why this might be the case? Is there anything else I should be checking?
Following a restart due to maintenance, the system stopped reporting this problem.
I didn't have chance to restart HDFS or CM in order to see which part was the problem.
HDFS fsck only checks the files that are persisted on hdfs and not open files. Since you were seeing just one missing block in the UI warnings of CM and NN and no missing blocks in fsck output, this could indicate that the missing block alert is being generated from a file that is open in the memory.
fsck option "-openforwrite" should show if the above is the case. Just a reference in case you hit the issue again.
I recently found a bug where Cloudera Manager and NameNode Web UI would report "bogus" missing blocks https://issues.apache.org/jira/browse/HDFS-13999 (Bogus missing block warning if the file is under construction when NN starts). But in fact hdfs fsck doesn't show any missing blocks/files.
The fix is not yet in any of the CDH releases. If the symptom described in the jira matches what you see, feel free to ignore it.