Support Questions
Find answers, ask questions, and share your expertise

Kafka python heartbeat timing out in session.timeout.ms

Kafka python heartbeat timing out in session.timeout.ms

Contributor

I found this https://stackoverflow.com/questions/43991845/kafka10-1-heartbeat-interval-ms-session-timeout-ms-and-... and it gives the description between the 3 but I don't quite understand why I'm still getting timeouts. I don't understand it in the context of my code why I have to increase the session_timeout_ms. I would expect to have to change the max_poll_interval_ms.

I have a consumer...

def my_consumer(self):
  consumer = KafkaConsumer("topic", bootstrap_servers=self.kafka_server + ":" + str(self.kafka_port),
  enable_auto_commit=False,
  group_id="topic_group",
  session_timeout_ms=300000)
  for msg in consumer:
     dict = json.loads(msg.value.decode('UTF-8'))
     self.lots_of_work(dict["id"]) //this can last for 5-15 minutes
     consumer.commit()



I start up my consumer, and it starts working on stuff. It will process a few messages and then when it runs into a message that runs over the default 10s session_timeout_ms it blows an error stating its getting kicked out of group. If I increase the session_timeout_ms it works (until it goes past the new longer timeout). The heartbeat thread should be in its own thread started by the KafkaConsumer call is it not? Where is the heartbeat thread being started and maintained? What can I check to figure out why the heartbeat is timing out? I could completely understand increasing the max_poll_interval_ms as thats "my" thread (e.g. the lots_of_work), but I don't quite get why the session_timeout_ms would need to be changed.