Newer versions of Storm use Netty client library instead of an older mechanism (ZeroMQ) for establishing connections between Nimbus manager and Supervisor instances. These connections are critical especially if a Storm executor task dies, in order to re-launch a new task to take the place of the dead one. Storm tasks can die for several reasons: OutOfMemory errors, RuntimeExceptions, errors accessing storage, etc. Storm provides auto-recovery and fault tolerance through Nimbus by monitoring the running executors, and sending a heartbeat request to determine the state of the task (alive or dead). The Netty protocol is supposed to discard any unread messages following timeout, and that appears to be happening in this case. Normally, this Netty error is a symptom of some other root cause. Can you check the storm logs (under /var/log/storm) and look for a root cause, perhaps an OutOfMemory error in one of the topology-specific logs on port 6700 or 6701. It is normal for Nimbus to re-establish Netty connections in this scenario. Problems re-connecting can be due to using the wrong hostname or IP address (NIC) for Nimbus, which is provided by parameter "nimbus.host" or "nimbus.seed" (depending on your version of Storm). Make sure you configure Storm Nimbus and Supervisors to use the same hostname/server/IP address.
