question Re: ambari showing corrupt blocks but not fsck in Support Questions

ambari showing corrupt blocks but not fsck

anshul_sisodia — Wed, 16 Dec 2015 17:42:25 GMT

Ambari UI shows corrupt blocks, fsck output shows not corrupt blocks and filesystem under / as healthy. Also when i run dfsadmin report it shows 'blocks with corrupt replicas ' same number as ambari showing

Re: ambari showing corrupt blocks but not fsck

nsabharwal — Wed, 16 Dec 2015 19:54:45 GMT

@Anshul Sisodia Please provide log file entries or snapshot

Re: ambari showing corrupt blocks but not fsck

pardeep_kumar — Wed, 16 Dec 2015 20:58:24 GMT

@Anshul Sisodia Could you please share FSCK, dfsadmin reports along with Ambari-server snapshot where it shows corrupt blocks?

Re: ambari showing corrupt blocks but not fsck

anshul_sisodia — Thu, 17 Dec 2015 15:52:50 GMT

DFSadmin report

Configured Capacity: 107909857935360 (98.14 TB) Present Capacity: 107907646763463 (98.14 TB) DFS Remaining: 63959036886116 (58.17 TB) DFS Used: 43948609877347 (39.97 TB) DFS Used%: 40.73% Under replicated blocks: 35016 Blocks with corrupt replicas: 113 Missing blocks: 0

Datanodes available: 3 (3 total, 0 dead) Live datanodes:

Name: x.x.x.x4 Hostname: x.x.x.x4 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 15310914557283 (13.93 TB) Non DFS Used: 821825081 (783.75 MB) DFS Remaining: 20658216262756 (18.79 TB) DFS Used%: 42.57% DFS Remaining%: 57.43% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 14 Last contact: Wed Dec 16 20:17:25 AEDT 2015

Name: x.x.x.x6 Hostname: x.x.x.x6 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 14348334051328 (13.05 TB) Non DFS Used: 497512448 (474.46 MB) DFS Remaining: 21621121081344 (19.66 TB) DFS Used%: 39.89% DFS Remaining%: 60.11% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1025 Last contact: Wed Dec 16 20:17:23 AEDT 2015

Name: x.x.x.x5 Hostname: x.x.x.x5 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 14289361268736 (13.00 TB) Non DFS Used: 891834368 (850.52 MB) DFS Remaining: 21679699542016 (19.72 TB) DFS Used%: 39.73% DFS Remaining%: 60.27% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1025

Re: ambari showing corrupt blocks but not fsck

anshul_sisodia — Thu, 17 Dec 2015 16:00:09 GMT

fsck output

......Status: HEALTHY Total size: 15311550475135 B (Total open files size: 36 B) Total dirs: 543341 Total files: 1572526 Total symlinks: 0 (Files currently being written: 5) Total blocks (validated): 1635306 (avg. block size 9363110 B) (Total open file blocks (not validated): 4) Minimally replicated blocks: 1635306 (99.99999 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 43091 (2.635042 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9589655 Corrupt blocks: 0 Missing replicas: 67060 (1.3669327 %) Number of data-nodes: 3 Number of racks: 1

Re: ambari showing corrupt blocks but not fsck

aervits — Thu, 24 Dec 2015 23:22:31 GMT

here are some additional flags you can pass to list the corrupt blocks. You might also be experiencing stale alerts on Ambari. To get more insight, drill down to the Ambari database, specifically in the alerts tables.

pathStart checking from this path.-deleteDelete corrupted files.-filesPrint out files being checked.-files -blocksPrint out the block report-files -blocks -locationsPrint out locations for every block.-files -blocks -racksPrint out network topology for data-node locations.-includeSnapshotsInclude snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it.-list-corruptfileblocksPrint out list of missing blocks and files they belong to.-moveMove corrupted files to /lost+found.-openforwritePrint out files opened for write.

Re: ambari showing corrupt blocks but not fsck

chri — Mon, 19 Aug 2019 12:34:18 GMT

Hi,

I had a similar effect after upgrading to Ambari 2.2.1.1 and HDP 2.4. HDFS Summary dashboard showed corrupt blocks

but fsck reported a heathly filesystem.

Status: HEALTHY
 Total size:    42317678169599 B (Total open files size: 89891159 B)
 Total dirs:    99625
 Total files:   289448
 Total symlinks:                0 (Files currently being written: 24)
 Total blocks (validated):      519195 (avg. block size 81506328 B) (Total open file blocks (not validated): 24)
 Minimally replicated blocks:   519195 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0056088
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          9
 Number of racks:               1
FSCK ended at Mon Apr 25 17:13:18 CEST 2016 in 4920 milliseconds
The filesystem under path '/' is HEALTHY

Solution: Restarted Ambari (server and agents) as well as Ambari Metrics. After a while, Ambari displayed 0 block errors which is in line with the output from fsck.

Cheers, Chris

Re: ambari showing corrupt blocks but not fsck

jaimin — Thu, 28 Apr 2016 03:41:01 GMT

@Anshul Sisodia

This was a bug in Ambari and, it has been reported and fixed via https://issues.apache.org/jira/browse/AMBARI-12931.

Fix will be a part of upcoming Ambari-2.2.2 release version.

Re: ambari showing corrupt blocks but not fsck

bjorn_olsen — Tue, 10 Jul 2018 13:03:07 GMT

In 2018 I'm having a similar issue - missing block alert displayed on Ambari although the fsck is displaying the file system as healthy.

This support article came up first on Google but the below support article resolved my issue. Linking for the benefit of others.

https://community.hortonworks.com/content/supportkb/185879/missing-block-alert-displayed-on-ambari-although-t.html

Re: ambari showing corrupt blocks but not fsck

snow1665437332_ — Mon, 19 Nov 2018 10:32:50 GMT

Our Ambari version is 2.5.1, but this problem still occurs: after adding a new data disk on the datanode, there is a corrupt blocks[1] on the ambari UI, but hdfs fsck / (even with the -includeSnapshot option) still shows There is no broken block, we restarted the ambari server, the ambari agent, the ambari metrics, still useless. Please help, thank you very much.