Skip to main content
roshanbi
Gold ⭐️
Jun 29, 2022 12:32 AM UTC
Question

Kafka checkpoint

Hello Team,

 

we are doing CDC by pushing data to Kafka and another pipeline will be reading data from Kafka. Whenever we restart the second pipeline (read from Kafka to Kudu), I notice there are thousands of records coming.

 

I would like to know how Kafka keeps the checkpoints? Is there any setting to change it?

 

Thanks,

Roshan

2 replies

araujo
Platinum ⭐️⭐️⭐️
Platinum ⭐️⭐️⭐️
Jun 29, 2022 8:41 AM UTC

@roshanbi ,

 

You must configure your Kafka consumer to use a consumer group and enable offset commits. This way the client will periodically save the last read offset internally in Kafka so that it can pick up from where it left off upon restarts.

 

Please check the Kafka documentation for the meaning of the properties below:

  • group.id

  • enable.auto.commit

  • auto.offset.reset

Cheers,

André

 

Translate
-- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs up button.
DianaTorres
Community Manager
Community Manager
Jul 1, 2022 5:02 PM UTC

@roshanbi Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks

Translate
Regards, Diana Torres, Senior Community Moderator