- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
ambari showing corrupt blocks but not fsck
- Labels:
-
Apache Ambari
-
Apache Hadoop
Created ‎12-16-2015 09:42 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ambari UI shows corrupt blocks, fsck output shows not corrupt blocks and filesystem under / as healthy. Also when i run dfsadmin report it shows 'blocks with corrupt replicas ' same number as ambari showing
Created ‎04-27-2016 08:41 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This was a bug in Ambari and, it has been reported and fixed via https://issues.apache.org/jira/browse/AMBARI-12931.
Fix will be a part of upcoming Ambari-2.2.2 release version.
Created ‎12-16-2015 11:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Anshul Sisodia Please provide log file entries or snapshot
Created ‎12-16-2015 12:58 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Anshul Sisodia Could you please share FSCK, dfsadmin reports along with Ambari-server snapshot where it shows corrupt blocks?
Created ‎12-17-2015 07:52 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
DFSadmin report
Configured Capacity: 107909857935360 (98.14 TB) Present Capacity: 107907646763463 (98.14 TB) DFS Remaining: 63959036886116 (58.17 TB) DFS Used: 43948609877347 (39.97 TB) DFS Used%: 40.73% Under replicated blocks: 35016 Blocks with corrupt replicas: 113 Missing blocks: 0
Datanodes available: 3 (3 total, 0 dead) Live datanodes:
Name: x.x.x.x4 Hostname: x.x.x.x4 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 15310914557283 (13.93 TB) Non DFS Used: 821825081 (783.75 MB) DFS Remaining: 20658216262756 (18.79 TB) DFS Used%: 42.57% DFS Remaining%: 57.43% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 14 Last contact: Wed Dec 16 20:17:25 AEDT 2015
Name: x.x.x.x6 Hostname: x.x.x.x6 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 14348334051328 (13.05 TB) Non DFS Used: 497512448 (474.46 MB) DFS Remaining: 21621121081344 (19.66 TB) DFS Used%: 39.89% DFS Remaining%: 60.11% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1025 Last contact: Wed Dec 16 20:17:23 AEDT 2015
Name: x.x.x.x5 Hostname: x.x.x.x5 Decommission Status : Normal Configured Capacity: 35969952645120 (32.71 TB) DFS Used: 14289361268736 (13.00 TB) Non DFS Used: 891834368 (850.52 MB) DFS Remaining: 21679699542016 (19.72 TB) DFS Used%: 39.73% DFS Remaining%: 60.27% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1025
Created ‎12-17-2015 08:00 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
fsck output
......Status: HEALTHY Total size: 15311550475135 B (Total open files size: 36 B) Total dirs: 543341 Total files: 1572526 Total symlinks: 0 (Files currently being written: 5) Total blocks (validated): 1635306 (avg. block size 9363110 B) (Total open file blocks (not validated): 4) Minimally replicated blocks: 1635306 (99.99999 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 43091 (2.635042 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9589655 Corrupt blocks: 0 Missing replicas: 67060 (1.3669327 %) Number of data-nodes: 3 Number of racks: 1
Created ‎12-24-2015 03:22 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
here are some additional flags you can pass to list the corrupt blocks. You might also be experiencing stale alerts on Ambari. To get more insight, drill down to the Ambari database, specifically in the alerts tables.
pathStart checking from this path.-deleteDelete corrupted files.-filesPrint out files being checked.-files -blocksPrint out the block report-files -blocks -locationsPrint out locations for every block.-files -blocks -racksPrint out network topology for data-node locations.-includeSnapshotsInclude snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it.-list-corruptfileblocksPrint out list of missing blocks and files they belong to.-moveMove corrupted files to /lost+found.-openforwritePrint out files opened for write.Created on ‎04-26-2016 05:28 AM - edited ‎08-19-2019 05:34 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I had a similar effect after upgrading to Ambari 2.2.1.1 and HDP 2.4. HDFS Summary dashboard showed corrupt blocks
but fsck reported a heathly filesystem.
Status: HEALTHY Total size: 42317678169599 B (Total open files size: 89891159 B) Total dirs: 99625 Total files: 289448 Total symlinks: 0 (Files currently being written: 24) Total blocks (validated): 519195 (avg. block size 81506328 B) (Total open file blocks (not validated): 24) Minimally replicated blocks: 519195 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0056088 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 9 Number of racks: 1 FSCK ended at Mon Apr 25 17:13:18 CEST 2016 in 4920 milliseconds The filesystem under path '/' is HEALTHY
Solution: Restarted Ambari (server and agents) as well as Ambari Metrics. After a while, Ambari displayed 0 block errors which is in line with the output from fsck.
Cheers, Chris
Created ‎04-27-2016 08:41 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This was a bug in Ambari and, it has been reported and fixed via https://issues.apache.org/jira/browse/AMBARI-12931.
Fix will be a part of upcoming Ambari-2.2.2 release version.
Created ‎07-10-2018 06:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In 2018 I'm having a similar issue - missing block alert displayed on Ambari although the fsck is displaying the file system as healthy.
This support article came up first on Google but the below support article resolved my issue. Linking for the benefit of others.
Created ‎11-19-2018 02:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Our Ambari version is 2.5.1, but this problem still occurs: after adding a new data disk on the datanode, there is a corrupt blocks[1] on the ambari UI, but hdfs fsck / (even with the -includeSnapshot option) still shows There is no broken block, we restarted the ambari server, the ambari agent, the ambari metrics, still useless. Please help, thank you very much.
