@EmilKle FYI ➤This issue typically arises because the comma (,) is a reserved delimiter in HBase for the hbase:meta table structure, used to separate the Table Name, Start Key, and Region ID. When a rowkey is inserted with an unexpected comma, the HBase shell and client API often misinterpret the entry as a malformed region name, causing GET or DELETE commands to route incorrectly or fail validation.   ➤Here is how you can approach a safe repair, as traditional methods like HBCK2 fixMeta often bypass these "illegal" keys if they don't follow the expected region naming convention.   1. Use the HBase Shell with Hexadecimal Rowkeys Standard string-based commands in the shell often fail because the shell parses the comma as a delimiter. Instead, find the exact byte representation of the rowkey and delete it using hexadecimal notation.   1. Find the Hex Key: Run a scan to get the exact bytes of the corrupted row.   scan 'hbase:meta', {ROWPREFIXFILTER => 'rowkey,'}   2. Delete using the binary string: If the rowkey is exactly rowkey,, use the binary notation in the shell:   delete 'hbase:meta', "rowkey\x2C", 'info:regioninfo' (Note: \x2C is the hex code for a comma). ... View more

@scala_ FYI ➤ It appears you have performed an exhaustive verification of the standard Kerberos and HBase configurations. The "GSS initiate failed" error in a Kerberized HBase environment, especially when standard connectivity and ticket validation pass, often points to subtle mismatches in how the Java process handles the security handshake or how the underlying OS interacts with the Kerberos libraries. ➤ Based on the logs and environment details you provided, here are the most likely remaining causes for this issue: 1. Java Cryptography Extension (JCE) and Encryption Types While you confirmed support for AES256 in krb5.conf, the Java Runtime Environment (JRE) itself may be restricting it. -The Issue: Older versions of Java 8 require the JCE Unlimited Strength Jurisdiction Policy Files to be manually installed to handle 256-bit encryption. If the Master is sending an AES256 ticket but the RegionServer's JVM is restricted, the GSS initiation will fail. -The Fix: Ensure the JCE policy files are installed, or if using a modern OpenJDK, ensure the java.security file allows all encryption strengths. You can also try restricting permitted_enctypes in krb5.conf to aes128-cts-hmac-sha1-96 temporarily to see if the connection succeeds with a lower bit-rate. 2. Reverse DNS (RDNS) Mismatch Kerberos is extremely sensitive to how hostnames are resolved. -The Issue: Even with entries in /etc/hosts, Java's GSSAPI often performs a reverse DNS lookup on the Master's IP. If the IP 10.51.39.121 (from your previous logs) resolves to a different hostname (or no hostname at all) than what is in your keytab (host117), the "GSS initiate" will fail. -The Fix: Add rdns = false to the [libdefaults] section of your /etc/krb5.conf on all nodes. This forces Kerberos to use the hostname provided by the application rather than trying to resolve the IP back to a name. 3. Service Principal Name (SPN) Case Sensitivity In hbase-site.xml, the principals are often defined with _HOST placeholders. -The Issue: If hbase.master.kerberos.principal is set to hbase/_HOST@REALM, HBase replaces _HOST with the fully qualified domain name (FQDN). If your system reports the FQDN as host117.kfs.local but the Kerberos Database (KDB) only has hbase/host117@REALM, the handshake fails. -The Fix: Ensure the output of the hostname -f command exactly matches the principal stored in the keytab. 4. JAAS "Server" vs. "Client" Sections Your earlier logs mentioned: “Added the Server login module in the JAAS config file.” -The Issue: In HBase, the RegionServer acts as a Client when connecting to the Master. If your JAAS configuration only has a Server section and is missing a Client section (or if the Client section has incorrect keytab details), the RegionServer will fail to initiate the GSS context toward the Master. -The Fix: Ensure your JAAS file contains both sections, and that the Client section points to the correct RegionServer keytab/principal. ... View more

@rizalt FYI ➤It appears your cluster is experiencing a Quorum Failure, where the critical metadata services (NameNode, JournalNodes, and ZooKeeper) are losing the ability to maintain a majority when one Data Center (DC) goes offline. ➤ Analyzing Your Failure In a High Availability (HA) setup, the NameNodes rely on a "Quorum" of JournalNodes (JN) and ZooKeeper (ZK) nodes to stay alive. If you have 5 JNs, you must have at least 3 running for either NameNode to function. Based on your diagram: The Problem: If you split your 5 JournalNodes evenly (e.g., 2 in DC1 and 3 in DC2), and the DC with 3 JNs goes down, the remaining 2 JNs cannot form a quorum. This causes both NameNodes to shut down immediately to prevent data corruption ("Split Brain"). DataNodes vs. NameNodes: In HDFS, the number of DataNodes (DN) that fail has no direct impact on whether the NameNode stays "Up" or "Down." You could lose 15 out of 16 DataNodes, and the NameNode should still stay active. The fact that your NameNodes are crashing when 8 DNs (one full server/DC) go down proves that your Quorum nodes (JN/ZK) are failing, not the DataNodes. ➤ The Solution: Quorum Placement To survive a full Data Center failure (50% of your physical infrastructure), you cannot rely on an even split of nodes. You need a third location (a "Witness" site) or an asymmetric distribution. 1. The 3-Site Strategy (Recommended) To handle a 1-DC failure with 5 JournalNodes and 5 ZooKeeper nodes, place them as follows: -DC 1: 2 JN, 2 ZK -DC 2: 2 JN, 2 ZK -Site 3 (Witness): 1 JN, 1 ZK (This can be a very small virtual machine or cloud instance). Why this works: If DC1 or DC2 fails, the remaining site + the Witness site equals 3 nodes, which satisfies the quorum ($3 > 5/2$). 2. Maximum DataNode Failure Theoretically: You can lose all but 1 DataNode ($N-1$), and the NameNode will stay "Active." Practically: If you have a replication factor of 3, and you lose 50% of your nodes, many blocks will become "Under-replicated," and some may become "Missing" if all three copies were in the DC that died. Solution: Ensure your Rack Awareness is configured so HDFS knows which nodes belong to which DC. This forces HDFS to keep at least one copy of data in each DC. ➤ Why 11 JN and 11 ZK didn't work Increasing the number of nodes to 11 actually makes the cluster more fragile if they are only placed in two locations. With 11 nodes, you need 6 to be alive to form a quorum. If you have 5 in DC1 and 6 in DC2, and DC2 fails, the 5 remaining nodes in DC1 cannot reach the 6-node requirement. ➤ Checklist for Survival Reduce to 5 JNs and 5 ZKs: Too many nodes increase network latency and management overhead. Add a 3rd Location: Even a single low-power node in a different building or cloud region to act as the "tie-breaker." Check dfs.namonode.edits.dir: Ensure the NameNodes are configured to point to all JournalNodes by URI. ZooKeeper FC: Ensure DFS ZK Failover Controller is running on both NameNode hosts. ... View more

@allen_chu FYI ➤ This issue—characterized by high CPU usage, a large number of threads stuck in DataXceiver, and a high load average—is a classic symptom of TCP socket leakage or connection hanging within the HDFS Data Transfer Protocol. ➤ Based on your top output and jstack, here is the detailed breakdown of what is happening and how to resolve it. ➤ Analysis of the Symptoms 1. CPU Saturation (99% per thread): Your top output shows dozens of DataXceiver threads consuming nearly 100% CPU each. This usually indicates that the threads are in a "busy-wait" or spinning state within the NIO epollWait call. 2. Stuck in epollWait: The jstack shows threads sitting in sun.nio.ch.EPollArrayWrapper.epollWait. While this is a normal state for a thread waiting for I/O, in your case, these threads are likely waiting for a packet from a client that has already disconnected or is "half-closed," but the DataNode hasn't timed out the connection. 3. Thread Exhaustion: With 792 threads, your DataNode is approaching its default dfs.datanode.max.transfer.threads limit (usually 4096, but often throttled by OS ulimit). As these threads accumulate, the DataNode loses the ability to accept new I/O requests, becoming unresponsive. ➤ Recommended Solutions 1. Increase Socket Timeouts (Immediate Fix) The most common cause is that the DataNode waits too long for a slow or dead client. You should tighten the transfer timeouts to force these "zombie" threads to close. => Update your hdfs-site.xml: dfs.datanode.socket.write.timeout: Default is often 0 (no timeout) or several minutes. Set this to 300000 (5 minutes). dfs.datanode.socket.reuse.keepalive: Set to true to allow better connection management. dfs.datanode.transfer.socket.send.buffer.size & recv.buffer.size: Ensure these are set to 131072 (128KB) to optimize throughput and prevent stalls. 2. Increase the Max Receiver Threads If your cluster handles high-concurrency workloads (like Spark or HBase), the default thread count might be too low. <property> <name>dfs.datanode.max.transfer.threads</name> <value>16384</value> </property> 3. Check for Network "Half-Closed" Connections Since the threads are stuck in read, it is possible the OS is keeping sockets in CLOSE_WAIT or FIN_WAIT2 states. a.] Check socket status: Run netstat -anp | grep 9866 | awk '{print $6}' | sort | uniq -c. b.] OS Tuning: Adjust the Linux kernel to more aggressively close dead connections. Add these to /etc/sysctl.conf: net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_intvl = 60 net.ipv4.tcp_keepalive_probes = 20 4. Address HDFS-14569 (Software Bug) Hadoop 3.1.1 is susceptible to a known issue where DataXceiver threads can leak during block moves or heavy balancer activity. Issue: DataXceiver fails to exit if a client stops sending data mid-packet but keeps the TCP connection open. Recommendation: If possible, upgrade to Hadoop 3.2.1+ or 3.3.x. These versions contain significantly improved NIO handling and better logic for terminating idle Xceivers. ➤ Diagnostic Step: Finding the "Bad" Clients To identify which clients are causing this, run this command on the DataNode: . netstat -atp | grep DataXceiver | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr This will tell you which IP addresses are holding the most DataXceiver connections. If one specific IP (like a single Spark executor or a specific user's edge node) has hundreds of connections, that client's code is likely not closing DFSClient instances correctly. ... View more