question Re: In Nifi is there anyway to consume kafka messages from beginning like we do from kafka console consumer or from next offset which last consumed? in Archives of Support Questions (Read Only)

In Nifi is there anyway to consume kafka messages from beginning like we do from kafka console consumer or from next offset which last consumed?

bigspark — Fri, 15 Jul 2016 16:25:23 GMT

In Nifi is there anyway to consume kafka messages from beginning like we do from kafka console consumer? Apart from consuming messages when Nifi has started it will receive current messages which are produced after the application has started but if i need to consume messages while my application was offline or from next offset which last consumed? Thank you

Re: In Nifi is there anyway to consume kafka messages from beginning like we do from kafka console consumer or from next offset which last consumed?

sball — Sun, 17 Jul 2016 17:37:23 GMT

There is a property on the GetKafka processor which species the offset to start reading from. "Auto Offset Reset" should be set to "smallest" to provide the equivalent of --from-beginning in the console consumer.

Re: In Nifi is there anyway to consume kafka messages from beginning like we do from kafka console consumer or from next offset which last consumed?

bigspark — Tue, 26 Jul 2016 18:15:44 GMT

@Simon Elliston BallDoes Nifi store offsets in memory as well. Just wondering how will it make sure not to read same offsets or duplicate messages from kafka? thanks