Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

ambari showing corrupt blocks but not fsck

avatar
Expert Contributor

Ambari UI shows corrupt blocks, fsck output shows not corrupt blocks and filesystem under / as healthy. Also when i run dfsadmin report it shows 'blocks with corrupt replicas ' same number as ambari showing

1 ACCEPTED SOLUTION

avatar
Super Collaborator

@Anshul Sisodia

This was a bug in Ambari and, it has been reported and fixed via https://issues.apache.org/jira/browse/AMBARI-12931.

Fix will be a part of upcoming Ambari-2.2.2 release version.

View solution in original post

9 REPLIES 9

avatar
Master Mentor

@Anshul Sisodia Please provide log file entries or snapshot

avatar

@Anshul Sisodia Could you please share FSCK, dfsadmin reports along with Ambari-server snapshot where it shows corrupt blocks?

avatar
Expert Contributor

DFSadmin report

Configured Capacity: 107909857935360 (98.14 TB) Present Capacity: 107907646763463 (98.14 TB) DFS Remaining: 63959036886116 (58.17 TB) DFS Used: 43948609877347 (39.97 TB) DFS Used%: 40.73% Under replicated blocks: 35016 Blocks with corrupt replicas: 113 Missing blocks: 0

Datanodes available: 3 (3 total, 0 dead) Live datanodes:

Name: x.x.x.x4 Hostname: x.x.x.x4 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 15310914557283 (13.93 TB) Non DFS Used: 821825081 (783.75 MB) DFS Remaining: 20658216262756 (18.79 TB) DFS Used%: 42.57% DFS Remaining%: 57.43% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 14 Last contact: Wed Dec 16 20:17:25 AEDT 2015

Name: x.x.x.x6 Hostname: x.x.x.x6 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 14348334051328 (13.05 TB) Non DFS Used: 497512448 (474.46 MB) DFS Remaining: 21621121081344 (19.66 TB) DFS Used%: 39.89% DFS Remaining%: 60.11% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1025 Last contact: Wed Dec 16 20:17:23 AEDT 2015

Name: x.x.x.x5 Hostname: x.x.x.x5 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 14289361268736 (13.00 TB) Non DFS Used: 891834368 (850.52 MB) DFS Remaining: 21679699542016 (19.72 TB) DFS Used%: 39.73% DFS Remaining%: 60.27% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1025

avatar
Expert Contributor

fsck output

......Status: HEALTHY Total size: 15311550475135 B (Total open files size: 36 B) Total dirs: 543341 Total files: 1572526 Total symlinks: 0 (Files currently being written: 5) Total blocks (validated): 1635306 (avg. block size 9363110 B) (Total open file blocks (not validated): 4) Minimally replicated blocks: 1635306 (99.99999 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 43091 (2.635042 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9589655 Corrupt blocks: 0 Missing replicas: 67060 (1.3669327 %) Number of data-nodes: 3 Number of racks: 1


ambari.png

avatar
Master Mentor

here are some additional flags you can pass to list the corrupt blocks. You might also be experiencing stale alerts on Ambari. To get more insight, drill down to the Ambari database, specifically in the alerts tables.

pathStart checking from this path.-deleteDelete corrupted files.-filesPrint out files being checked.-files -blocksPrint out the block report-files -blocks -locationsPrint out locations for every block.-files -blocks -racksPrint out network topology for data-node locations.-includeSnapshotsInclude snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it.-list-corruptfileblocksPrint out list of missing blocks and files they belong to.-moveMove corrupted files to /lost+found.-openforwritePrint out files opened for write.

avatar
Rising Star

Hi,

I had a similar effect after upgrading to Ambari 2.2.1.1 and HDP 2.4. HDFS Summary dashboard showed corrupt blocks

3688-hdfs-summary.png

but fsck reported a heathly filesystem.

Status: HEALTHY
 Total size:    42317678169599 B (Total open files size: 89891159 B)
 Total dirs:    99625
 Total files:   289448
 Total symlinks:                0 (Files currently being written: 24)
 Total blocks (validated):      519195 (avg. block size 81506328 B) (Total open file blocks (not validated): 24)
 Minimally replicated blocks:   519195 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0056088
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          9
 Number of racks:               1
FSCK ended at Mon Apr 25 17:13:18 CEST 2016 in 4920 milliseconds
The filesystem under path '/' is HEALTHY

Solution: Restarted Ambari (server and agents) as well as Ambari Metrics. After a while, Ambari displayed 0 block errors which is in line with the output from fsck.

Cheers, Chris

avatar
Super Collaborator

@Anshul Sisodia

This was a bug in Ambari and, it has been reported and fixed via https://issues.apache.org/jira/browse/AMBARI-12931.

Fix will be a part of upcoming Ambari-2.2.2 release version.

avatar
Contributor

In 2018 I'm having a similar issue - missing block alert displayed on Ambari although the fsck is displaying the file system as healthy.

This support article came up first on Google but the below support article resolved my issue. Linking for the benefit of others.

https://community.hortonworks.com/content/supportkb/185879/missing-block-alert-displayed-on-ambari-a...

avatar

Our Ambari version is 2.5.1, but this problem still occurs: after adding a new data disk on the datanode, there is a corrupt blocks[1] on the ambari UI, but hdfs fsck / (even with the -includeSnapshot option) still shows There is no broken block, we restarted the ambari server, the ambari agent, the ambari metrics, still useless. Please help, thank you very much.