Support Questions
Find answers, ask questions, and share your expertise

Kafka streams (0.10) not "at-least-once" guarantee in HDP 2.5.3?

Kafka streams (0.10) not "at-least-once" guarantee in HDP 2.5.3?

Expert Contributor

Hello I am using HDP with kafka / kafka-streams 0.10 (2.10-0.10.0)

What I have found is if I have an input and output topic, and for some reason the output topic is not available (e.g. it is not created yet!) then the kakfa-streams just continue consuming messages from input and logging error, without actually stopping or any indication of issues with publishing to output topics. (What I would expect is the stream stop and on restart it will try from the message that was not published to down stream.)

Surely this violate "at-least-once"? I suppose this is just due to the version of kafka-streams is fairly old and newer version (say 0.10.2) would have this fixed in later HDP / Kafka update?