Created on
06-22-2023
10:08 AM
- edited on
06-27-2023
04:42 AM
by
VidyaSargur
High level Problem Explanation
The occurrence of the "block missing" and "block corruption" alerts is observed in the Cloudera Manager dashboard or the namenode webui. While there isn't a common cause, these alerts can be due to any of the outlined issues.
Detailed Technical Explanation
When the namenode does not receive reports for all of the block replicas of a file, it marks those blocks as missing. Similarly, when all the block replicas are corrupted, the namenode marks them as corrupt. However, if at least one of the block replica(1/3) is reported as being healthy, then there is no missing or corruption alert.
There are multiple scenarios in which blocks can be marked as missing and corrupt.
Scenario 1
During a major upgrade, users may encounter block missing errors. This issue arises due to insufficient datanode heap.
The upgrade process involves migrating the disk layout and requires additional datanode heap space. users who are upgrading to releases CDP 7.1.7 and above will not experience these issues. However, for other users, it is necessary to increase the datanode heap size by 2x from current heap. Once the upgrade is completed, the heap size can be reverted to its original value.
The adjustment in heap size is required specifically for the layout upgrade from -56 (256x256) to -57 (32x32). During this process, each volume's blocks are scanned, and a list of LinkArgs and file objects are stored as String which resulting in significant heap utilization.
Example of file object which stored as String,
/data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta
/data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825
If a datanode crashes with an Out of Memory error during the upgrade, then you experience a missing blocks. This occurs because the blocks were in the old layout, but the datanode assumes the blocks were in the new layout. If you encounter this issue, it is recommended to contact Cloudera for assistance in fixing the block layout.
Scenario 2
User manually replaces the old fsimage in the namenode's current directory, which resulted inconsistent behavior and shows missing and corrupt blocks after reboot.
Scenario 3
In rare instances, users may encounter a scenario where the dfs.checksum.type is set to NULL, so disabling the safety mechanisms in place results block corruption.
HDFS incorporates a built-in checksum mechanism that is enabled by default. When a client writes data, it calculates a CRC checksum for the data. Additionally, the last datanode in the write pipeline also calculates a checksum for the received data. If the checksums from the client and datanode match, the data is considered valid and is stored on disk along with the checksum information.
Checksums play a crucial role in identifying various hardware issues, including problems with network cables, incompatible MTU network settings, and faulty disks or controllers. It is highly recommended to enable checksums on an HDFS cluster to ensure data integrity.
However, when the checksum.type is set to NULL, all these safety mechanisms are effectively disabled. Without checksums, HDFS is unable to detect corruption. This means that data can be silently corrupted during tasks such as data migration between hosts or due to disk failures.
To identify this block corruption and checksum, you can use the "hdfs debug verify" command to verify the metadata of a block.
hdfs debug verify -meta <metadata-file> [-block <block-file>]
For assistance in efficiently checking the block checksums of the entire HDFS filesystem, we recommend reaching out to the Cloudera storage team. We have developed a specialized tool designed to perform this task quickly and effectively.
Scenario 4
When the size of a block report exceeds the limit set by "ipc.maximum.data.length" (default value of 64 MB), the namenode rejects those block reports which results missing blocks. This rejection is typically accompanied by log entries in both the namenode and datanode logs indicating that the "Protocol message was too large."
In scenarios where a single datanode holds over 15 million blocks, it becomes more likely to encounter block reports that surpass the configured limit. This can result in the rejection of such block reports by the namenode due to their excessive size.
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException in offerService org.apache.hadoop.ipc.RemoteException(java.io.IOException): java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException:
Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.runBlockOp(BlockManager.java:5174)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1574)
Scenario 5
There are instances where a user administrator may accidentally delete disk data with the intention of reclaiming disk space, but without a complete understanding of Hadoop data storage in the disk. This leads to the loss of blocks.
Scenario 6
The filesystem appears to be in a healthy state, but when running FSCK, it detects a missing block. This issue arises from a snapshot reference where some of its blocks are missing.
hdfs fsck / -includeSnapshots -list-corruptfileblocks
Example:
The problem arises at this stage: FSCK continues to identify file "f1" as corrupt even after reloading it with a new block.
This scenario presents a false positive, where FSCK should appropriately report missing references, whether they stem from live data or a snapshot. The issue described here HDFS-15770, a minor improvement that requires fix.
Scenario 7
The occurrence of a "Too many open file" exception in the DataNode can lead to block deletion, ultimately resulting in missing blocks. This issue has been addressed in the CDP.
Scenario 8
In a rare scenario that has not been conclusively proven yet, a user experienced a loss of multiple racks in their cluster due to rack failure. After restarting the DataNode following a power event, the blocks were found in the "rbw" (Replica Being Written) directory.
All the blocks from the "rbw" directory transitioned into the "RWR" (Replica Waiting to be Recovered) state. As a result, the NameNode marked these blocks as corrupt, citing the reason as "Reason.INVALID_STATE".
However, the issue lies in the fact that the NameNode only removes a replica from the corrupt map if the reason for corruption is "GENSTAMP_MISMATCH". In this particular scenario, the reason is different, causing a challenge in removing the replicas from the corrupt map.
Scenario 9
There is an issue where certain information in an incremental block report (IBR) gets replaced with existing stored information while queuing the block report on a standby Namenode. This can result in false block corruption reports.
In a specific incident, a Namenode transitioned to an active state and started reporting missing blocks with the reason "SIZE_MISMATCH" for corruption. However, upon investigation, it was discovered that the blocks were actually appended, and the sizes on the datanodes were correct.
The root cause was identified as the standby Namenode queuing IBRs with modified information.
Users experiencing this false block corruption issue are advised to run a manual block report, as it can help rectify the erroneous reports and restore accurate block information. This issues is addressed in CDP 7 SP1 and SP2.
hdfs dfsadmin -triggerBlockReport [-incremental] <datanode_host:ipc_port>
hdfs fsck / -list-corruptfileblocks -files -blocks -locations
Connecting to namenode via https://host:20102/fsck?ugi=hdfs&listcorruptfileblocks=1&files=1&blocks=1&locations=1&path=%2F
The filesystem under path '/' has 0 CORRUPT files
Note: A healthy filesystem should always have zero corrupt files.
hdfs dfsadmin -report >dfsadmin.report
cat dfsadmin.report | grep "Missing blocks" | grep "with replication factor 1"
dataLength + is longer than maximum configured RPC length " + maxDataLength + ". RPC came from " + getHostAddress();
find <dfs.datanode.data.dir> -type f -iname <blkid>*
Repeat the same step in all datanodes
Note Examine all the provided scenarios and their corresponding actions. It is important to note that if a block is physically present on any of the datanodes, it is generally recoverable, except in cases where the block is corrupt.