I've noticed in Ambari under HDFS metrics that we have 2 blocks with corrupt replicas.
Running " hdfs fsck / " shows no corrupt blocks and system is healthy.
Running "hdfs dfsadmin-report" shows 2 corrupt replicas (same as Ambari dashboard)
I've restarted Ambari metrics & Ambari Agents on all nodes + Ambari-server as noted in one of the threads i came across but still - problem remains.
Ambari is 2.5.2
Any ideas how to fix this issue ?
Below is the procedure to remove the corrupt blocks or files
Locate the files have blocks that are corrupt.
$ hdfs fsck / | egrep -v '^\.+
$ hdfs fsck hdfs://ip.or.host:50070/ | egrep -v '^\.+
/path/to/filename.file_extension: CORRUPT blockpool BP-1016133662-10.29.100.41-1415825958975 block blk_1073904305 /path/to/filename.file_extension: MISSING 1 blocks of total size 15620361 B
$ hdfs dfs -rm /path/to/filename.file_extension
$ hdfs dfs -rm -skipTrash /path/to/filename.file_extension
$ hdfs fsck /path/to/filename/file_extension -locations -blocks -files
$ hdfs fsck hdfs://ip.or.hostname.of.namenode:50070/path/to/filename/file_extension -locations -blocks -files