Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How does storm handle failed tuples?

Solved Go to solution
Highlighted

How does storm handle failed tuples?

Contributor

Storm Version: 0.10.0.2.4

Using a Kafka Spout.

How does storm handle failed tuples?

How many times will storm retry a failed tuple?

What frequency will storm retry the failed tuple?

What is the max tuple count a topology can handle between all spouts and bolts?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How does storm handle failed tuples?

Rising Star

Hi @Jon Maestas

Answering your questions inline:

How does storm handle failed tuples?

When you are using at least once processing (acking and anchoring) is when Storm will handle tuple failures by retries. Retry means re-emitting a tuple from Spout.

How many times will storm retry a failed tuple?

This depends on the Spout's logic, in case of Kafka Spout for 0.10.x Storm there's the ability for exponential backoff retry (https://github.com/apache/storm/blob/0.10.x-branch/external/storm-kafka/src/jvm/storm/kafka/ExponentialBackoffMsgRetryManager.java)

What frequency will storm retry the failed tuple?

ExponentialBackoff will determine the frequency.

What is the max tuple count a topology can handle between all spouts and bolts?

I am guessing you are asking for maximum number of tuples at any given point can be in Storm's buffers? This = Bolt Count * Executor Count * TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE + Bolt Count * Executor Count * TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE

You can find out the value of these buffers from Ambari -> Storm -> Config -> Search "buffer"

Please note that the above is theoretical maximum, Max Spout Pending (topology.max.spout.pending) throttles the number of in-flight tuples from the Spout.

There's also transfer buffers which will add bit more to the above calculated number.

Please refer to Michael Noll's blog for more details about Storm Buffers (http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/)

Hope this answers your questions.

View solution in original post

3 REPLIES 3

Re: How does storm handle failed tuples?

Rising Star

Hi @Jon Maestas

Answering your questions inline:

How does storm handle failed tuples?

When you are using at least once processing (acking and anchoring) is when Storm will handle tuple failures by retries. Retry means re-emitting a tuple from Spout.

How many times will storm retry a failed tuple?

This depends on the Spout's logic, in case of Kafka Spout for 0.10.x Storm there's the ability for exponential backoff retry (https://github.com/apache/storm/blob/0.10.x-branch/external/storm-kafka/src/jvm/storm/kafka/ExponentialBackoffMsgRetryManager.java)

What frequency will storm retry the failed tuple?

ExponentialBackoff will determine the frequency.

What is the max tuple count a topology can handle between all spouts and bolts?

I am guessing you are asking for maximum number of tuples at any given point can be in Storm's buffers? This = Bolt Count * Executor Count * TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE + Bolt Count * Executor Count * TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE

You can find out the value of these buffers from Ambari -> Storm -> Config -> Search "buffer"

Please note that the above is theoretical maximum, Max Spout Pending (topology.max.spout.pending) throttles the number of in-flight tuples from the Spout.

There's also transfer buffers which will add bit more to the above calculated number.

Please refer to Michael Noll's blog for more details about Storm Buffers (http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/)

Hope this answers your questions.

View solution in original post

Highlighted

Re: How does storm handle failed tuples?

Rising Star

@Jon Maestas Please accept the answer if this answered your questions.

Highlighted

Re: How does storm handle failed tuples?

Good write-up from @Ambud Sharma plus you can visit http://storm.apache.org/releases/1.0.2/Guaranteeing-message-processing.html for info from the source. Additionally, take a peek at the picture below I just exported from our http://hortonworks.com/training/class/hdp-developer-storm-and-trident-fundamentals/ course that might help visualize all of this information.

11589-acker.jpg

Good luck and happy Storming!

Don't have an account?
Coming from Hortonworks? Activate your account here