question Kafka checkpoint in Support Questions

Kafka checkpoint

roshanbi — Wed, 29 Jun 2022 00:32:03 GMT

Hello Team,

we are doing CDC by pushing data to Kafka and another pipeline will be reading data from Kafka. Whenever we restart the second pipeline (read from Kafka to Kudu), I notice there are thousands of records coming.

I would like to know how Kafka keeps the checkpoints? Is there any setting to change it?

Thanks,

Roshan

Re: Kafka checkpoint

araujo — Wed, 29 Jun 2022 08:42:26 GMT

@roshanbi ,

You must configure your Kafka consumer to use a consumer group and enable offset commits. This way the client will periodically save the last read offset internally in Kafka so that it can pick up from where it left off upon restarts.

Please check the Kafka documentation for the meaning of the properties below:

group.id
enable.auto.commit
auto.offset.reset

Cheers,

André

Re: Kafka checkpoint

DianaTorres — Fri, 01 Jul 2022 17:02:27 GMT

@roshanbi Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks