Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to know the reason for missing blocks

Solved Go to solution

how to know the reason for missing blocks

we have ambari cluster with 3 masters machine and 4 datanode machine

we run the hdfs dfsadmin -report

and we found Missing blocks: 4

how to know the reason for these missing blocks?

second what is the workaround that we need to do regarding that?

hdfs dfsadmin -report
Configured Capacity: 8226130288640 (7.48 TB)
Present Capacity: 8225526102776 (7.48 TB)
DFS Remaining: 8205621209848 (7.46 TB)
DFS Used: 19904892928 (18.54 GB)
DFS Used%: 0.24%
Under replicated blocks: 4
Blocks with corrupt replicas: 0
Missing blocks: 4
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (4):
Name: 10.164.252.32:50010 (worker03.sys76.com)
Hostname: worker03.sys76.com
Decommission Status : Normal
Configured Capacity: 1170504683520 (1.06 TB)
DFS Used: 5715611648 (5.32 GB)
Non DFS Used: 0 (0 B)
DFS Remaining: 1164727208338 (1.06 TB)
DFS Used%: 0.49%
DFS Remaining%: 99.51%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 6
Last contact: Wed Jan 31 13:34:02 UTC 2018
Name: 10.164.252.33:50010 (worker04.sys76.com)
Hostname: worker04.sys76.com
Decommission Status : Normal
Configured Capacity: 2351875026944 (2.14 TB)
DFS Used: 4573270016 (4.26 GB)
Non DFS Used: 0 (0 B)
DFS Remaining: 2347124950656 (2.13 TB)
DFS Used%: 0.19%
DFS Remaining%: 99.80%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 6
Last contact: Wed Jan 31 13:34:02 UTC 2018
Name: 10.164.252.31:50010 (worker02.sys76.com)
Hostname: worker02.sys76.com
Decommission Status : Normal
Configured Capacity: 2351875026944 (2.14 TB)
DFS Used: 5077798912 (4.73 GB)
Non DFS Used: 0 (0 B)
DFS Remaining: 2346627408110 (2.13 TB)
DFS Used%: 0.22%
DFS Remaining%: 99.78%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 6
Last contact: Wed Jan 31 13:34:02 UTC 2018
Name: 10.164.252.30:50010 (worker01.sys76.com)
Hostname: worker01.sys76.com
Decommission Status : Normal
Configured Capacity: 2351875551232 (2.14 TB)
DFS Used: 4538212352 (4.23 GB)
Non DFS Used: 0 (0 B)
DFS Remaining: 2347141642744 (2.13 TB)
DFS Used%: 0.19%
DFS Remaining%: 99.80%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 8
Last contact: Wed Jan 31 13:34:02 UTC 2018
Michael-Bronson
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: how to know the reason for missing blocks

Guru

@Michael Bronson , Missing data block can be related to data corruption. Use 'hdfs fsck <path> -list-corruptfileblocks -files -locations' to find out which replicas got corrupted.

Secondly, In order to fix issue, you can delete the corrupted blocks using 'hdfs fsck / -delete'

I hope you find below thread useful for handing missing blocks.

https://community.hortonworks.com/questions/17917/best-way-of-handling-corrupt-or-missing-blocks.htm...

1 REPLY 1
Highlighted

Re: how to know the reason for missing blocks

Guru

@Michael Bronson , Missing data block can be related to data corruption. Use 'hdfs fsck <path> -list-corruptfileblocks -files -locations' to find out which replicas got corrupted.

Secondly, In order to fix issue, you can delete the corrupted blocks using 'hdfs fsck / -delete'

I hope you find below thread useful for handing missing blocks.

https://community.hortonworks.com/questions/17917/best-way-of-handling-corrupt-or-missing-blocks.htm...

Don't have an account?
Coming from Hortonworks? Activate your account here