Your OS seems to be running out of free port numbers in the ephemeral range. Typically on Linux this is in range 32k to 64k, which is quite a lot of ports. A common reason is abuse of software clients (due to excessive connections being created without use of shared connection pools, or a leak of connections due to non-closure in the code), or lower level problems with the socket closure (such as the FIN stage of TCP not being correctly processed, causing the OS to hold the port open for an extended period of time waiting for the final close to complete). Are you perhaps executing a lot of concurrent programs on your cluster, or use a multi-threaded app that builds a new network client (for HDFS, etc.) under each thread? When you experience this, you could run an lsof check on the host of the failing task to find which PID(s) are occupying most of the network client ephemeral ports and if there is a pattern to their destination(s). This can help figure out where the problem specifically lies, and what category (in the above) it may belong to.
... View more
