ambari showing corrupt blocks but not fsck

[#1]

Ambari UI shows corrupt blocks, fsck output shows not corrupt blocks and filesystem under / as healthy. Also when i run dfsadmin report it shows 'blocks with corrupt replicas ' same number as ambari showing

[#2]

@Anshul Sisodia Please provide log file entries or snapshot

[#3]

@Anshul Sisodia Could you please share FSCK, dfsadmin reports along with Ambari-server snapshot where it shows corrupt blocks?

[#4]

DFSadmin report

Configured Capacity: 107909857935360 (98.14 TB) Present Capacity: 107907646763463 (98.14 TB) DFS Remaining: 63959036886116 (58.17 TB) DFS Used: 43948609877347 (39.97 TB) DFS Used%: 40.73% Under replicated blocks: 35016 Blocks with corrupt replicas: 113 Missing blocks: 0

Datanodes available: 3 (3 total, 0 dead) Live datanodes:

Name: x.x.x.x4 Hostname: x.x.x.x4 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 15310914557283 (13.93 TB) Non DFS Used: 821825081 (783.75 MB) DFS Remaining: 20658216262756 (18.79 TB) DFS Used%: 42.57% DFS Remaining%: 57.43% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 14 Last contact: Wed Dec 16 20:17:25 AEDT 2015

Name: x.x.x.x6 Hostname: x.x.x.x6 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 14348334051328 (13.05 TB) Non DFS Used: 497512448 (474.46 MB) DFS Remaining: 21621121081344 (19.66 TB) DFS Used%: 39.89% DFS Remaining%: 60.11% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1025 Last contact: Wed Dec 16 20:17:23 AEDT 2015

Name: x.x.x.x5 Hostname: x.x.x.x5 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 14289361268736 (13.00 TB) Non DFS Used: 891834368 (850.52 MB) DFS Remaining: 21679699542016 (19.72 TB) DFS Used%: 39.73% DFS Remaining%: 60.27% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1025

[#5]

fsck output

......Status: HEALTHY Total size: 15311550475135 B (Total open files size: 36 B) Total dirs: 543341 Total files: 1572526 Total symlinks: 0 (Files currently being written: 5) Total blocks (validated): 1635306 (avg. block size 9363110 B) (Total open file blocks (not validated): 4) Minimally replicated blocks: 1635306 (99.99999 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 43091 (2.635042 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9589655 Corrupt blocks: 0 Missing replicas: 67060 (1.3669327 %) Number of data-nodes: 3 Number of racks: 1

[#6]

here are some additional flags you can pass to list the corrupt blocks. You might also be experiencing stale alerts on Ambari. To get more insight, drill down to the Ambari database, specifically in the alerts tables.

pathStart checking from this path.-deleteDelete corrupted files.-filesPrint out files being checked.-files -blocksPrint out the block report-files -blocks -locationsPrint out locations for every block.-files -blocks -racksPrint out network topology for data-node locations.-includeSnapshotsInclude snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it.-list-corruptfileblocksPrint out list of missing blocks and files they belong to.-moveMove corrupted files to /lost+found.-openforwritePrint out files opened for write.

[#7]

Hi,

I had a similar effect after upgrading to Ambari 2.2.1.1 and HDP 2.4. HDFS Summary dashboard showed corrupt blocks

but fsck reported a heathly filesystem.

Status: HEALTHY
 Total size:    42317678169599 B (Total open files size: 89891159 B)
 Total dirs:    99625
 Total files:   289448
 Total symlinks:                0 (Files currently being written: 24)
 Total blocks (validated):      519195 (avg. block size 81506328 B) (Total open file blocks (not validated): 24)
 Minimally replicated blocks:   519195 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0056088
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          9
 Number of racks:               1
FSCK ended at Mon Apr 25 17:13:18 CEST 2016 in 4920 milliseconds
The filesystem under path '/' is HEALTHY

Solution: Restarted Ambari (server and agents) as well as Ambari Metrics. After a while, Ambari displayed 0 block errors which is in line with the output from fsck.

Cheers, Chris

[#8 · Accepted solution]

@Anshul Sisodia

This was a bug in Ambari and, it has been reported and fixed via https://issues.apache.org/jira/browse/AMBARI-12931.

Fix will be a part of upcoming Ambari-2.2.2 release version.

[#9]

In 2018 I'm having a similar issue - missing block alert displayed on Ambari although the fsck is displaying the file system as healthy.

This support article came up first on Google but the below support article resolved my issue. Linking for the benefit of others.

https://community.hortonworks.com/content/supportkb/185879/missing-block-alert-displayed-on-ambari-a...

[#10]

Our Ambari version is 2.5.1, but this problem still occurs: after adding a new data disk on the datanode, there is a corrupt blocks[1] on the ambari UI, but hdfs fsck / (even with the -includeSnapshot option) still shows There is no broken block, we restarted the ambari server, the ambari agent, the ambari metrics, still useless. Please help, thank you very much.