Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is anyone else having errors with Netty using Storm?

avatar
Contributor

I'm getting errors from my spout that look like:

2015-12-03 08:30:57.425 b.s.m.n.Client [ERROR] discarding 1 messages because the Netty client to Netty-Client-storm03/XXX.XXX.XXX.XXX:6700 is being closed

Is this an error with Netty or something to do with my stack? We're on HDP 2.3.2

Adam

1 ACCEPTED SOLUTION

avatar

Newer versions of Storm use Netty client library instead of an older mechanism (ZeroMQ) for establishing connections between Nimbus manager and Supervisor instances. These connections are critical especially if a Storm executor task dies, in order to re-launch a new task to take the place of the dead one. Storm tasks can die for several reasons: OutOfMemory errors, RuntimeExceptions, errors accessing storage, etc. Storm provides auto-recovery and fault tolerance through Nimbus by monitoring the running executors, and sending a heartbeat request to determine the state of the task (alive or dead). The Netty protocol is supposed to discard any unread messages following timeout, and that appears to be happening in this case. Normally, this Netty error is a symptom of some other root cause. Can you check the storm logs (under /var/log/storm) and look for a root cause, perhaps an OutOfMemory error in one of the topology-specific logs on port 6700 or 6701. It is normal for Nimbus to re-establish Netty connections in this scenario. Problems re-connecting can be due to using the wrong hostname or IP address (NIC) for Nimbus, which is provided by parameter "nimbus.host" or "nimbus.seed" (depending on your version of Storm). Make sure you configure Storm Nimbus and Supervisors to use the same hostname/server/IP address.

View solution in original post

11 REPLIES 11

avatar
Explorer

The netty error reporting in storm is very poor as all it tells you is that it cannot connect to something at a port or that it is dropping a message. This is problematic when trying to identify what is causing connection failures or lost messages.

avatar
New Contributor

connection attempt 1 to Netty-Client-testHost1/IP:6703 failed: java.lang.RuntimeException: Returned channel was actually not established

assist me with above error

,

Hi

I am getting below error on all workers logs

connection attempt 1 to Netty-Client-testHost1:6703 failed: java.lang.RuntimeException: Returned channel was actually not established

kindly assist with this issue