High level Problem Explanation The primary garbage collection challenge often arises when the heap configuration of the Namenode or Datanode is inadequate. GC pauses can trigger Namenode crash, failovers, and performance bottlenecks. In larger clusters, there is no response at the time of transitioning to an another Namenode. Detailed Technical Explanation The Namenode comprises three layers of management: Inode Management Block Management Snapshot Management All of these layers are stored in the Namenode in-memory heap, and any substantial modifications to them can lead to increased heap usage. Similarly, the Datanode also maintains block mapping information in its memory. Example Of Inode Reference   <inode> <id>26890</id> <type>DIRECTORY</type> <name>training</name> <mtime>1660512737451</mtime> <permission>hdfs:supergroup:0755</permission> <nsquota>-1</nsquota><dsquota>-1</dsquota> </inode> <inode> <id>26893</id> <type>FILE</type> <name>file1</name> <replication>3</replication> <mtime>1660512801514</mtime> <atime>1660512797820</atime> <preferredBlockSize>134217728</preferredBlockSize> <permission>hdfs:supergroup:0644</permission> <storagePolicyId>0</storagePolicyId> </inode>   Example Of Block Reference   <blocks> <block><id>1073751671</id><genstamp>10862</genstamp><numBytes>134217728</numBytes></block> <block><id>1073751672</id><genstamp>10863</genstamp><numBytes>134217728</numBytes></block> <block><id>1073751673</id><genstamp>10864</genstamp><numBytes>134217728</numBytes></block> <block><id>1073751674</id><genstamp>10865</genstamp><numBytes>134217728</numBytes></block> <block><id>1073751675</id><genstamp>10866</genstamp><numBytes>134217728</numBytes></block> <block><id>1073751676</id><genstamp>10867</genstamp><numBytes>134217728</numBytes></block> <block><id>1073751677</id><genstamp>10868</genstamp><numBytes>116452794</numBytes></block> </blocks>   Example Of Snapshot Reference   <SnapshotSection> <snapshotCounter>1</snapshotCounter> <numSnapshots>1</numSnapshots> <snapshottableDir> <dir>26890</dir> </snapshottableDir> <snapshot> <id>0</id> <root> <id>26890</id> <type>DIRECTORY</type> <name>snap1</name> <mtime>1660513260644</mtime> <permission>hdfs:supergroup:0755</permission> <nsquota>-1</nsquota> <dsquota>-1</dsquota> </root> </snapshot> </SnapshotSection>   High heap usage can be caused by several factors, including: Inadequate heap size for the current dataset. Lack of proper heap tuning configurations. Failure to include "-XX:CMSInitiatingOccupancyFraction" with the default value set at 92. Gradual growth in the number of inodes/files over time. Sudden spikes in HDFS data volumes. Sudden changes in HDFS snapshot activity. Job executions generating millions of small files. System slowness, including CPU or IO bottlenecks. Symptoms Namenode failure and Namenode transitions Namenode logs showing JVM metrics, with any pause exceeding 10 seconds being a concern.  WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 143694ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=143973ms      Occasionally, GC pauses are induced by system overhead factors, such as CPU spikes, IO contention, slow disk performance, or kernel delays. Look for the phrase "No GCs detected" in the Namenode log. WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 83241ms No GCs detected     Do not misinterpret the FSNamesystem lock message; the root cause of the issue is actually GC overhead the pause before this event.   INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of suppressed read-lock reports: 0 Longest read-lock held at xxxx for 143973ms via java.lang.Thread.getStackTrace(Thread.java:1559) org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1058) org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.readUnlock(FSNamesystemLock.java:187) org.apache.hadoop.hdfs.server.namenode.FSNamesystem.readUnlock(FSNamesystem.java:1684) org.apache.hadoop.hdfs.server.namenode.ContentSummaryComputationContext.yield(ContentSummaryComputationContext.java:134) ..   Relevant Data to Check To gain insights into usage patterns, it's advisable to regularly collect heap histograms. If you have enabled the parameters -XX:+PrintClassHistogramAfterFullGC and -XX:+PrintClassHistogramBeforeFullGC in the heap configuration, there's no need for additional histogram collection. Metrics:   Cloudera Manager -> Charts -> Chart Builder -> SELECT jvm_gc_rate WHERE roleType = NAMENODE and hostname = "<problamatic namenode hostname>" SELECT jvm_max_memory_mb, jvm_heap_used_mb WHERE roleType = NAMENODE and hostname = "<problamatic namenode hostname>"      Minor pauses are generally tolerable, but when pause times exceed the defined "ha.health-monitor.rpc-timeout.ms" timeout, the risk of a Namenode crash increases, particularly if the Namenode writes edits after a GC pause.  Also, Heap usage reaching 85% of the total heap can also lead to issues. Even slight pauses, such as 2-5 seconds occurring frequently, can pose problems. Resolution To adjust the heap size effectively, consider the current system load and anticipated future workloads. Given the diverse range of customer use cases, our standard guidelines may not always align perfectly with each scenario. Factors like differing file lengths, snapshot utilization, encryption zones, and more can influence requirements. Instead, it's advisable to analyze the current number of files, directories, and blocks in conjunction with the existing heap utilization. For instance, if you have 30 million files and directories, 40 million blocks, and 200 snapshots, and your current heap usage is 45GB out of a total heap size of 50GB, you can estimate the additional heap required for a 50% load increase. In this case, you would need an additional 23GB of heap (current_heap * additional_load_percentage/100) to accommodate the increased load. Please note that you can find the current heap usage metrics in the Namenode web UI or through CM (Cloudera Manager) charts by querying:  "SELECT jvm_max_memory_mb, jvm_heap_used_mb WHERE roleType = NAMENODE and hostname = "<problematic Namenode hostname>"​" To ensure smooth operation, it's wise to allocate an additional 10-15% of heap beyond the total configured heap. If Sentry is configured, consider adding a 10-20% buffer. It's crucial to maintain the total count of files and directories in the cluster below 350 million. Namenode performance tends to decline after surpassing this threshold. We generally recommend not exceeding 300 million files per cluster. Please cross-reference our guidance with official heap documents provided by CDP. If a customer has configured G1 Garbage Collection for the Namenode and there is no notable improvement in Namenode GC performance, consider using CMS.  Note, During our internal testing, we observed no significant performance improvement in the Namenode when using G1 Garbage Collection but leave it to your own testings. For Datanode performance, ensure that the Java heap is set to 1GB per 1 million blocks. Additionally, verify that all Datanodes are balanced, with a roughly equal distribution of blocks. Avoid scenarios where one node holds a disproportionately high number of blocks compared to others. ... View more

High level Problem Explanation The occurrence of the "block missing" and "block corruption" alerts is observed in the Cloudera Manager dashboard or the namenode webui. While there isn't a common cause, these alerts can be due to any of the outlined issues.   Detailed Technical Explanation When the namenode does not receive reports for all of the block replicas of a file, it marks those blocks as missing. Similarly, when all the block replicas are corrupted, the namenode marks them as corrupt. However, if at least one of the block replica(1/3) is reported as being healthy, then there is no missing or corruption alert. There are multiple scenarios in which blocks can be marked as missing and corrupt.   Scenario 1 During a major upgrade, users may encounter block missing errors. This issue arises due to insufficient datanode heap.   The upgrade process involves migrating the disk layout and requires additional datanode heap space. users who are upgrading to releases CDP 7.1.7 and above will not experience these issues. However, for other users, it is necessary to increase the datanode heap size by 2x from current heap. Once the upgrade is completed, the heap size can be reverted to its original value. The adjustment in heap size is required specifically for the layout upgrade from -56 (256x256) to -57 (32x32). During this process, each volume's blocks are scanned, and a list of LinkArgs and file objects are stored as String which resulting in significant heap utilization.   Example of file object which stored as String,   /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825_1001.meta /data01/dfs/dn/current/BP-586623041-127.0.0.1-1617017575175/current/finalized/subdir0/subdir0/blk_1073741825   If a datanode crashes with an Out of Memory error during the upgrade, then you experience a missing blocks. This occurs because the blocks were in the old layout, but the datanode assumes the blocks were in the new layout. If you encounter this issue, it is recommended to contact Cloudera for assistance in fixing the block layout.   Scenario 2 User manually replaces the old fsimage in the namenode's current directory, which resulted  inconsistent behavior and shows missing and corrupt blocks after reboot.    Scenario 3 In rare instances, users may encounter a scenario where the dfs.checksum.type is set to NULL, so disabling the safety mechanisms in place results block corruption.   HDFS incorporates a built-in checksum mechanism that is enabled by default. When a client writes data, it calculates a CRC checksum for the data. Additionally, the last datanode in the write pipeline also calculates a checksum for the received data. If the checksums from the client and datanode match, the data is considered valid and is stored on disk along with the checksum information.   Checksums play a crucial role in identifying various hardware issues, including problems with network cables, incompatible MTU network settings, and faulty disks or controllers. It is highly recommended to enable checksums on an HDFS cluster to ensure data integrity.   However, when the checksum.type is set to NULL, all these safety mechanisms are effectively disabled. Without checksums, HDFS is unable to detect corruption. This means that data can be silently corrupted during tasks such as data migration between hosts or due to disk failures.    To identify this block corruption and checksum, you can use the "hdfs debug verify" command to verify the metadata of a block.   hdfs debug verify -meta <metadata-file> [-block <block-file>]   For assistance in efficiently checking the block checksums of the entire HDFS filesystem, we recommend reaching out to the Cloudera storage team. We have developed a specialized tool designed to perform this task quickly and effectively.   Scenario 4 When the size of a block report exceeds the limit set by "ipc.maximum.data.length" (default value of 64 MB), the namenode rejects those block reports which results missing blocks. This rejection is typically accompanied by log entries in both the namenode and datanode logs indicating that the "Protocol message was too large."   In scenarios where a single datanode holds over 15 million blocks, it becomes more likely to encounter block reports that surpass the configured limit. This can result in the rejection of such block reports by the namenode due to their excessive size.     WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException in offerService org.apache.hadoop.ipc.RemoteException(java.io.IOException): java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.runBlockOp(BlockManager.java:5174) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1574)   Scenario 5 There are instances where a user administrator may accidentally delete disk data with the intention of reclaiming disk space, but without a complete understanding of Hadoop data storage in the disk. This leads to the loss of blocks. Scenario 6  The filesystem appears to be in a healthy state, but when running FSCK, it detects a missing block. This issue arises from a snapshot reference where some of its blocks are missing.   hdfs fsck / -includeSnapshots -list-corruptfileblocks   Example: A file named "f1" is created, and its corresponding block ID is "blk_100005". A snapshot named "s1" is taken for "f1". Running FSCK shows that the filesystem is healthy, as expected. However, at a later point, "blk_100005" goes missing due to unknown reasons. Consequently, the filesystem becomes corrupted, and FSCK accurately reports the corrupt filename and its block. The next step involves removing the corrupt file "f1" using the command: hdfs dfs -rm -skipTrash f1. Create or reload a new file, "f1", which creates a new block labeled "blk_100006". The problem arises at this stage: FSCK continues to identify file "f1" as corrupt even after reloading it with a new block. This scenario presents a false positive, where FSCK should appropriately report missing references, whether they stem from live data or a snapshot. The issue described here HDFS-15770, a minor improvement that requires fix.   Scenario 7 The occurrence of a "Too many open file" exception in the DataNode can lead to block deletion, ultimately resulting in missing blocks. This issue has been addressed in the CDP.   Scenario 8 In a rare scenario that has not been conclusively proven yet, a user experienced a loss of multiple racks in their cluster due to rack failure. After restarting the DataNode following a power event, the blocks were found in the "rbw" (Replica Being Written) directory.   All the blocks from the "rbw" directory transitioned into the "RWR" (Replica Waiting to be Recovered) state. As a result, the NameNode marked these blocks as corrupt, citing the reason as "Reason.INVALID_STATE".   However, the issue lies in the fact that the NameNode only removes a replica from the corrupt map if the reason for corruption is "GENSTAMP_MISMATCH". In this particular scenario, the reason is different, causing a challenge in removing the replicas from the corrupt map.   Scenario 9 There is an issue where certain information in an incremental block report (IBR) gets replaced with existing stored information while queuing the block report on a standby Namenode. This can result in false block corruption reports.   In a specific incident, a Namenode transitioned to an active state and started reporting missing blocks with the reason "SIZE_MISMATCH" for corruption. However, upon investigation, it was discovered that the blocks were actually appended, and the sizes on the datanodes were correct.   The root cause was identified as the standby Namenode queuing IBRs with modified information.   Users experiencing this false block corruption issue are advised to run a manual block report, as it can help rectify the erroneous reports and restore accurate block information. This issues is addressed in CDP 7 SP1 and SP2.    hdfs dfsadmin -triggerBlockReport [-incremental] <datanode_host:ipc_port>   Verification of Problem As a first step, it is important to verify if the fsck command reports any corrupt blocks. hdfs fsck / -list-corruptfileblocks -files -blocks -locations Connecting to namenode via https://host:20102/fsck?ugi=hdfs&listcorruptfileblocks=1&files=1&blocks=1&locations=1&path=%2F The filesystem under path '/' has 0 CORRUPT files              Note: A healthy filesystem should always have zero corrupt files. Determine the number of missing blocks that have only one replica. These blocks are of critical concern as the chances of recovery are minimal. hdfs dfsadmin -report >dfsadmin.report cat dfsadmin.report | grep "Missing blocks" | grep "with replication factor 1"     Verify the operational status of all datanodes in the cluster. If three or more datanodes are offline or not functioning correctly, it can result in missing and corrupt blocks. Ensure that you check for any dead datanodes in the cluster. If three or more datanodes are in a dead state, it can lead to the presence of missing and corrupt blocks. Verify if there are multiple disk failures across the datanodes. In such cases, the presence of missing and corrupt blocks is expected. Check whether the storage capacity of the datanode is fully utilized, reaching 100%. In such instances, the datanode may become unresponsive and fail to send block reports. Verify if the block report sent by the datanode is rejected by the namenode. This can be observed in the namenode logs as a warning or error message. dataLength + is longer than maximum configured RPC length " + maxDataLength + ". RPC came from " + getHostAddress();     Confirm if the user has made changes to the node configuration groups in Cloudera Manager (CM). Such modifications could potentially result in the omission of several data disk directories, leading to issues with missing data. Verify whether the block still exists in the local filesystem or if it has been unintentionally removed by users. find <dfs.datanode.data.dir> -type f -iname <blkid>*     Repeat the same step in all datanodes Ensure that you check if the mount point has been unmounted due to a filesystem failure. It is important to verify the datanode and system logs for any relevant information in this regard. Verify whether the block has been written into the root volume as a result of disk auto unmount. It is essential to consider that the data might be concealed if you remount the filesystem on top of the existing datanode directory. Note Examine all the provided scenarios and their corresponding actions. It is important to note that if a block is physically present on any of the datanodes, it is generally recoverable, except in cases where the block is corrupt.     ... View more

We have seen performance issue and stability issue in heavy workload system, this is due to disk latency/Shared Disk. I.e frequent namenode failover, longer boot time, slower checkpoint, slower logging, Higher fsync will cause session expiry, etc. We never recommend a shared disk for Namenode, Journal node and Zookeeper. All of these services should have a dedicated disk. You can configure following disk type according to your HDFS workload. [dfs.namenode.name.dir] Namenode fsimage directory, dedicated disk => HDD 15K RPM [dfs.journalnode.edits.dir] Namenode edit log directory, dedicated disk => SSD [dataDir from zoo.cfg] Zoookeeper snapshot and transaction logs, for normal usage from ZKFC and HBase => HDD 15K RPM Zoookeeper snapshot and transaction logs, If it used by Nifi/ Accumulo/ Kafka/ Storm/ HBase and ZKFC => SSD If you are using RAID for meta-directory(dfs.namenode.name.dir & dfs.journalnode.edits.dir), then disable RAID and check the Non-RAID performance. There is a strong redundant for meta-directory (fsimage & edits are available from Standby NameNode and remaining QJNs). If RAID is not disabled for JN, then consider using different RAID. RAID 1 and RAID 10 are also good for the dfs.journalnode.edits.dir set rather than RAID 5, due to increased in write latency for small block writes. if you don't have a faster disk, then don't consider using fsimage replication. It will impact write performance even if one of the disks slower. ... View more

The service RPC port gives the DataNodes a dedicated port to report their status via block reports and heartbeats. The port is also used by Zookeeper Failover Controllers for periodic health checks by the automatic failover logic. The port is never used by client applications hence it reduces RPC queue contention between client requests and DataNode messages. Steps: 1) Ambari -> HDFS -> Configs -> Advanced -> Custom hdfs-site -> Add Property dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn1=<namenode1 host:rpc port> dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn2=<namenode2 host:rpc port> dfs.namenode.service.handler.count=(dfs.namenode.handler.count / 2) This RPC port receives all DN and ZKFC requests like block report, heartbeat, liveness report, etc.. Example from hdfs-site.xml, dfs.nameservices=shva dfs.internal.nameservices=shva dfs.ha.namenodes.shva=nn1,nn2 dfs.namenode.rpc-address.shva.nn1=hwxunsecure2641.openstacklocal:8020 dfs.namenode.rpc-address.shva.nn2=hwxunsecure2642.openstacklocal:8020 dfs.namenode.handler.count=200 Service RPC host, port and handler threads: dfs.namenode.servicerpc-address.shva.nn1=hwxunsecure2641.openstacklocal:8040 dfs.namenode.servicerpc-address.shva.nn2=hwxunsecure2642.openstacklocal:8040 dfs.namenode.service.handler.count=100 2) Restart Standby Namenode. You must wait till Standby Namenode out of safemode. Note: You can check Safemode status in Standby Namenode UI. 3) Restart Active Namenode. 4) Stop Standby Namenode ZKFC controller. 5) Stop Active Namenode ZKFC controller. 6) Login Active Namenode and reset Namenode HA state. #su - hdfs $hdfs zkfc -formatZK 7) Login Standby Namenode and reset Namenode HA state. #su - hdfs $hdfs zkfc -formatZK 😎 Start Active Namenode ZKFC controller. 9) Start Standby Namenode ZKFC controller. 10) Rolling restart the Datanodes. Note: Please check, Nodemanager should not be installed in Namenode box because it uses same port 8040. If installed then you need to change service RPC port from 8040 to different port. Ref: scaling-the-hdfs-namenode ... View more