@allen_chu FYI
➤ This issue—characterized by high CPU usage, a large number of threads stuck in DataXceiver, and a high load average—is a classic symptom of TCP socket leakage or connection hanging within the HDFS Data Transfer Protocol.
➤ Based on your top output and jstack, here is the detailed breakdown of what is happening and how to resolve it.
➤ Analysis of the Symptoms
1. CPU Saturation (99% per thread): Your top output shows dozens of DataXceiver threads consuming nearly 100% CPU each. This usually indicates that the threads are in a "busy-wait" or spinning state within the NIO epollWait call.
2. Stuck in epollWait: The jstack shows threads sitting in sun.nio.ch.EPollArrayWrapper.epollWait. While this is a normal state for a thread waiting for I/O, in your case, these threads are likely waiting for a packet from a client that has already disconnected or is "half-closed," but the DataNode hasn't timed out the connection.
3. Thread Exhaustion: With 792 threads, your DataNode is approaching its default dfs.datanode.max.transfer.threads limit (usually 4096, but often throttled by OS ulimit). As these threads accumulate, the DataNode loses the ability to accept new I/O requests, becoming unresponsive.
➤ Recommended Solutions
1. Increase Socket Timeouts (Immediate Fix)
The most common cause is that the DataNode waits too long for a slow or dead client. You should tighten the transfer timeouts to force these "zombie" threads to close.
=> Update your hdfs-site.xml:
dfs.datanode.socket.write.timeout: Default is often 0 (no timeout) or several minutes. Set this to 300000 (5 minutes).
dfs.datanode.socket.reuse.keepalive: Set to true to allow better connection management.
dfs.datanode.transfer.socket.send.buffer.size & recv.buffer.size: Ensure these are set to 131072 (128KB) to optimize throughput and prevent stalls.
2. Increase the Max Receiver Threads
If your cluster handles high-concurrency workloads (like Spark or HBase), the default thread count might be too low.
<property>
<name>dfs.datanode.max.transfer.threads</name>
<value>16384</value>
</property>
3. Check for Network "Half-Closed" Connections
Since the threads are stuck in read, it is possible the OS is keeping sockets in CLOSE_WAIT or FIN_WAIT2 states.
a.] Check socket status: Run netstat -anp | grep 9866 | awk '{print $6}' | sort | uniq -c.
b.] OS Tuning: Adjust the Linux kernel to more aggressively close dead connections. Add these to /etc/sysctl.conf:
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 20
4. Address HDFS-14569 (Software Bug)
Hadoop 3.1.1 is susceptible to a known issue where DataXceiver threads can leak during block moves or heavy balancer activity.
Issue: DataXceiver fails to exit if a client stops sending data mid-packet but keeps the TCP connection open.
Recommendation: If possible, upgrade to Hadoop 3.2.1+ or 3.3.x. These versions contain significantly improved NIO handling and better logic for terminating idle Xceivers.
➤ Diagnostic Step: Finding the "Bad" Clients
To identify which clients are causing this, run this command on the DataNode:
.
netstat -atp | grep DataXceiver | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
This will tell you which IP addresses are holding the most DataXceiver connections. If one specific IP (like a single Spark executor or a specific user's edge node) has hundreds of connections, that client's code is likely not closing DFSClient instances correctly.
