@EmilKle FYI ➤This issue typically arises because the comma (,) is a reserved delimiter in HBase for the hbase:meta table structure, used to separate the Table Name, Start Key, and Region ID. When a rowkey is inserted with an unexpected comma, the HBase shell and client API often misinterpret the entry as a malformed region name, causing GET or DELETE commands to route incorrectly or fail validation.   ➤Here is how you can approach a safe repair, as traditional methods like HBCK2 fixMeta often bypass these "illegal" keys if they don't follow the expected region naming convention.   1. Use the HBase Shell with Hexadecimal Rowkeys Standard string-based commands in the shell often fail because the shell parses the comma as a delimiter. Instead, find the exact byte representation of the rowkey and delete it using hexadecimal notation.   1. Find the Hex Key: Run a scan to get the exact bytes of the corrupted row.   scan 'hbase:meta', {ROWPREFIXFILTER => 'rowkey,'}   2. Delete using the binary string: If the rowkey is exactly rowkey,, use the binary notation in the shell:   delete 'hbase:meta', "rowkey\x2C", 'info:regioninfo' (Note: \x2C is the hex code for a comma). ... View more

@scala_ FYI ➤ It appears you have performed an exhaustive verification of the standard Kerberos and HBase configurations. The "GSS initiate failed" error in a Kerberized HBase environment, especially when standard connectivity and ticket validation pass, often points to subtle mismatches in how the Java process handles the security handshake or how the underlying OS interacts with the Kerberos libraries. ➤ Based on the logs and environment details you provided, here are the most likely remaining causes for this issue: 1. Java Cryptography Extension (JCE) and Encryption Types While you confirmed support for AES256 in krb5.conf, the Java Runtime Environment (JRE) itself may be restricting it. -The Issue: Older versions of Java 8 require the JCE Unlimited Strength Jurisdiction Policy Files to be manually installed to handle 256-bit encryption. If the Master is sending an AES256 ticket but the RegionServer's JVM is restricted, the GSS initiation will fail. -The Fix: Ensure the JCE policy files are installed, or if using a modern OpenJDK, ensure the java.security file allows all encryption strengths. You can also try restricting permitted_enctypes in krb5.conf to aes128-cts-hmac-sha1-96 temporarily to see if the connection succeeds with a lower bit-rate. 2. Reverse DNS (RDNS) Mismatch Kerberos is extremely sensitive to how hostnames are resolved. -The Issue: Even with entries in /etc/hosts, Java's GSSAPI often performs a reverse DNS lookup on the Master's IP. If the IP 10.51.39.121 (from your previous logs) resolves to a different hostname (or no hostname at all) than what is in your keytab (host117), the "GSS initiate" will fail. -The Fix: Add rdns = false to the [libdefaults] section of your /etc/krb5.conf on all nodes. This forces Kerberos to use the hostname provided by the application rather than trying to resolve the IP back to a name. 3. Service Principal Name (SPN) Case Sensitivity In hbase-site.xml, the principals are often defined with _HOST placeholders. -The Issue: If hbase.master.kerberos.principal is set to hbase/_HOST@REALM, HBase replaces _HOST with the fully qualified domain name (FQDN). If your system reports the FQDN as host117.kfs.local but the Kerberos Database (KDB) only has hbase/host117@REALM, the handshake fails. -The Fix: Ensure the output of the hostname -f command exactly matches the principal stored in the keytab. 4. JAAS "Server" vs. "Client" Sections Your earlier logs mentioned: “Added the Server login module in the JAAS config file.” -The Issue: In HBase, the RegionServer acts as a Client when connecting to the Master. If your JAAS configuration only has a Server section and is missing a Client section (or if the Client section has incorrect keytab details), the RegionServer will fail to initiate the GSS context toward the Master. -The Fix: Ensure your JAAS file contains both sections, and that the Client section points to the correct RegionServer keytab/principal. ... View more

@rizalt FYI ➤It appears your cluster is experiencing a Quorum Failure, where the critical metadata services (NameNode, JournalNodes, and ZooKeeper) are losing the ability to maintain a majority when one Data Center (DC) goes offline. ➤ Analyzing Your Failure In a High Availability (HA) setup, the NameNodes rely on a "Quorum" of JournalNodes (JN) and ZooKeeper (ZK) nodes to stay alive. If you have 5 JNs, you must have at least 3 running for either NameNode to function. Based on your diagram: The Problem: If you split your 5 JournalNodes evenly (e.g., 2 in DC1 and 3 in DC2), and the DC with 3 JNs goes down, the remaining 2 JNs cannot form a quorum. This causes both NameNodes to shut down immediately to prevent data corruption ("Split Brain"). DataNodes vs. NameNodes: In HDFS, the number of DataNodes (DN) that fail has no direct impact on whether the NameNode stays "Up" or "Down." You could lose 15 out of 16 DataNodes, and the NameNode should still stay active. The fact that your NameNodes are crashing when 8 DNs (one full server/DC) go down proves that your Quorum nodes (JN/ZK) are failing, not the DataNodes. ➤ The Solution: Quorum Placement To survive a full Data Center failure (50% of your physical infrastructure), you cannot rely on an even split of nodes. You need a third location (a "Witness" site) or an asymmetric distribution. 1. The 3-Site Strategy (Recommended) To handle a 1-DC failure with 5 JournalNodes and 5 ZooKeeper nodes, place them as follows: -DC 1: 2 JN, 2 ZK -DC 2: 2 JN, 2 ZK -Site 3 (Witness): 1 JN, 1 ZK (This can be a very small virtual machine or cloud instance). Why this works: If DC1 or DC2 fails, the remaining site + the Witness site equals 3 nodes, which satisfies the quorum ($3 > 5/2$). 2. Maximum DataNode Failure Theoretically: You can lose all but 1 DataNode ($N-1$), and the NameNode will stay "Active." Practically: If you have a replication factor of 3, and you lose 50% of your nodes, many blocks will become "Under-replicated," and some may become "Missing" if all three copies were in the DC that died. Solution: Ensure your Rack Awareness is configured so HDFS knows which nodes belong to which DC. This forces HDFS to keep at least one copy of data in each DC. ➤ Why 11 JN and 11 ZK didn't work Increasing the number of nodes to 11 actually makes the cluster more fragile if they are only placed in two locations. With 11 nodes, you need 6 to be alive to form a quorum. If you have 5 in DC1 and 6 in DC2, and DC2 fails, the 5 remaining nodes in DC1 cannot reach the 6-node requirement. ➤ Checklist for Survival Reduce to 5 JNs and 5 ZKs: Too many nodes increase network latency and management overhead. Add a 3rd Location: Even a single low-power node in a different building or cloud region to act as the "tie-breaker." Check dfs.namonode.edits.dir: Ensure the NameNodes are configured to point to all JournalNodes by URI. ZooKeeper FC: Ensure DFS ZK Failover Controller is running on both NameNode hosts. ... View more

➤ It sounds like you are encountering a common issue in HDFS where the metadata overhead and block minimums are causing a massive discrepancy between your actual data size and your disk utilization.   While 650 files at 4MB each technically equals 2.6GB of data, the way HDFS manages these on your physical disks (especially in smaller or test clusters) can lead to unexpected storage consumption.   ➤ Root Causes of the 100% Utilization   1. Reserved Space and "Non-DFS Used" HDFS does not have access to the entire disk. By default, Hadoop reserves a portion of the disk for the OS and non-Hadoop data (usually defined by dfs.datanode.du.reserved). If you are running on small disks (e.g., 20GB–50GB), the combination of your data, logs, and reserved space can quickly hit the 100% threshold.   2. Local Filesystem Block Overheads Even though your HDFS block size is 4MB, your underlying OS filesystem (EXT4 or XFS) uses its own block size (usually 4KB). However, the metadata for 650 individual files, their checksums (.meta files), and the edit logs on the NameNode create a "death by a thousand cuts" scenario for small disks.   3. Log Accumulation Check /var/log/hadoop or your configured log directory. In HDFS 3.3.5, if a cluster is struggling with space, the DataNodes and NameNodes generate massive amounts of "Heartbeat" and "Disk Full" logs, which consume the remaining Non-DFS space, pushing the disk to 100%.   ➤How to Tackle the Situation Step 1: Identify Where the Space Is Going Run the following command to see if the space is taken by HDFS data or other files: $ hdfs dfsadmin -report   DFS Used: Space taken by your 650 files. Non-DFS Used: Space taken by logs, OS, and other applications. If this is high, your logs are the culprit.     Step 2: Clear Logs and Temporary Data If "Non-DFS Used" is high, clear out the Hadoop log directory:   # Example path rm -rf /var/log/hadoop/hdfs/*.log.* rm -rf /var/log/hadoop/hdfs/*.out.*   Step 3: Adjust the "Disk Checked" Thresholds By default, a DataNode stops working if the disk is 95% full. If you are in a test environment and need to squeeze out more space, you can lower the reserved space in hdfs-site.xml:   <property>   <name>dfs.datanode.du.reserved</name>   <value>1073741824</value>  </property>   Step 4: Combine Small Files (Long-term Fix) HDFS is designed for large files.1 650 files of 4MB are considered "Small Files."   The Problem: Every file, regardless of size, takes up roughly 150 bytes of RAM on the NameNode and creates separate metadata entries.   The Solution: Use the getmerge command or a MapReduce/Spark job to combine these 650 files into 2 or 3 larger files (e.g., 1GB each). ... View more

@G_B FYI ➤Based on your report, your data is physically safe on the disks, but the HDFS Metadata Link is broken because your DataNodes cannot talk to your NameNode properly. Here is the breakdown of why this is happening and how to fix it. 1. The Result: "Missing Blocks" and "Forbidden" Errors Missing Blocks (6303): Your NameNode knows the files should exist (metadata is loaded), but because the DataNode blockReport failed due to the Java error, the DataNodes haven't told the NameNode which blocks they are holding. Num of Blocks: 0: Look at your datanode dfsadmin report. It says Num of Blocks: 0. The NameNode thinks that node is empty because the Block Report failed. Head of file / Forbidden: Since the NameNode thinks there are 0 blocks available, it tells your client "I have no DataNodes to give you for this file." 2 . Restart all Hadoop services (NameNode first, then DataNodes). 3. Solution Step 2: Clear the "Lease" and Trigger Block Reports => Check the DataNode logs. You want to see: Successfully sent block report. => Once the reports are successful, run the report again: hdfs dfsadmin -report. => The "Missing Blocks" count should start dropping toward zero, and Num of Blocks should increase. 4.. Troubleshooting the "File Not Found" in fsck The reason fsck said the file didn't exist while ls showed it is likely due to NameNode Safe Mode. When a NameNode starts up and sees 6,000+ missing blocks, it often enters Safe Mode to prevent data loss. Check if you are in safe mode: hdfs dfsadmin -safemode get If it is ON, do not leave it manually until your DataNodes have finished reporting their blocks. ... View more

@allen_chu FYI ➤ This issue—characterized by high CPU usage, a large number of threads stuck in DataXceiver, and a high load average—is a classic symptom of TCP socket leakage or connection hanging within the HDFS Data Transfer Protocol. ➤ Based on your top output and jstack, here is the detailed breakdown of what is happening and how to resolve it. ➤ Analysis of the Symptoms 1. CPU Saturation (99% per thread): Your top output shows dozens of DataXceiver threads consuming nearly 100% CPU each. This usually indicates that the threads are in a "busy-wait" or spinning state within the NIO epollWait call. 2. Stuck in epollWait: The jstack shows threads sitting in sun.nio.ch.EPollArrayWrapper.epollWait. While this is a normal state for a thread waiting for I/O, in your case, these threads are likely waiting for a packet from a client that has already disconnected or is "half-closed," but the DataNode hasn't timed out the connection. 3. Thread Exhaustion: With 792 threads, your DataNode is approaching its default dfs.datanode.max.transfer.threads limit (usually 4096, but often throttled by OS ulimit). As these threads accumulate, the DataNode loses the ability to accept new I/O requests, becoming unresponsive. ➤ Recommended Solutions 1. Increase Socket Timeouts (Immediate Fix) The most common cause is that the DataNode waits too long for a slow or dead client. You should tighten the transfer timeouts to force these "zombie" threads to close. => Update your hdfs-site.xml: dfs.datanode.socket.write.timeout: Default is often 0 (no timeout) or several minutes. Set this to 300000 (5 minutes). dfs.datanode.socket.reuse.keepalive: Set to true to allow better connection management. dfs.datanode.transfer.socket.send.buffer.size & recv.buffer.size: Ensure these are set to 131072 (128KB) to optimize throughput and prevent stalls. 2. Increase the Max Receiver Threads If your cluster handles high-concurrency workloads (like Spark or HBase), the default thread count might be too low. <property> <name>dfs.datanode.max.transfer.threads</name> <value>16384</value> </property> 3. Check for Network "Half-Closed" Connections Since the threads are stuck in read, it is possible the OS is keeping sockets in CLOSE_WAIT or FIN_WAIT2 states. a.] Check socket status: Run netstat -anp | grep 9866 | awk '{print $6}' | sort | uniq -c. b.] OS Tuning: Adjust the Linux kernel to more aggressively close dead connections. Add these to /etc/sysctl.conf: net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_intvl = 60 net.ipv4.tcp_keepalive_probes = 20 4. Address HDFS-14569 (Software Bug) Hadoop 3.1.1 is susceptible to a known issue where DataXceiver threads can leak during block moves or heavy balancer activity. Issue: DataXceiver fails to exit if a client stops sending data mid-packet but keeps the TCP connection open. Recommendation: If possible, upgrade to Hadoop 3.2.1+ or 3.3.x. These versions contain significantly improved NIO handling and better logic for terminating idle Xceivers. ➤ Diagnostic Step: Finding the "Bad" Clients To identify which clients are causing this, run this command on the DataNode: . netstat -atp | grep DataXceiver | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr This will tell you which IP addresses are holding the most DataXceiver connections. If one specific IP (like a single Spark executor or a specific user's edge node) has hundreds of connections, that client's code is likely not closing DFSClient instances correctly. ... View more