Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What happens to the failed message in apache-storm-1.0.2

Highlighted

What happens to the failed message in apache-storm-1.0.2

New Contributor

In storm version apache-storm-1.0.2 , there a field called maxRetries in the retryservice provided for the kafkaSpout and when a bolt fails for > maxretries the message is acked by spout.So does this mean the message them is removed from the kafka queue. If so, with out processing a message we ae actually removing it? Isn't that bad?

3 REPLIES 3
Highlighted

Re: What happens to the failed message in apache-storm-1.0.2

Contributor

Hi Sushmitha,

The java docs on the KafkaSpout indicate that the fail method for the spout will re-queue the message for playback at a later time

'The tuple emitted by this spout with the msgId identifier has failed to be fully processed. Typically, an implementation of this method will put that message back on the queue to be replayed at a later time.'

but the code itself looks like it just acks the message in order to skip over further processing and no rescheduling takes place. Messages are never "removed" from the Kafka queue per-se, they just won't continue (to attempt) to be processed after it tries so many times unsuccessfully.

Highlighted

Re: What happens to the failed message in apache-storm-1.0.2

New Contributor

http://storm.apache.org/releases/1.0.0/Guaranteeing-message-processing.html.This article talks about removing the message from queue after getting acked.What does removing here mean? Like you have said, if the messages are just not replayed after maxRetries then since it is acked, the message is understood as processed by kafka and so it won't be consumed by another consumer.This leads to missing some messages.How is this dealt in 1.0.2 version of storm

Highlighted

Re: What happens to the failed message in apache-storm-1.0.2

New Contributor

@Ryan Templeton

Don't have an account?
Coming from Hortonworks? Activate your account here