Kafka partition leader election

[#1]

Hi,

We often see ​NOT_LEADER_FOR_PARTITION in kafka. On what basis does kafka leader election happen ? What are all the reasons that cause kafka partition leader election to trigger?

[#2]

Hi,

NOT_LEADER_FOR_PARTITION in kafka.

when a client is trying to access a topic partition for which the given broker is not the leader. For example a producer can only send new messages to the partition that is considered the leader for a replica set of partitions. Only after the leader partition receives the messages will they be sent to the other brokers that hold the replicas of that partition.

It can happen when the client/producer has stale information about which broker is the leader for a partition. In this case the NotLeaderForPartitionException is thrown by the broker that the client is connecting. Another possible reason , when a leader re-election happened recently (like when restarting brokers one by one in your case and for a short time the leader partitions for certain replicas were unavailable and new leaders were elected for the partitions that used to have a leader on the unavailable broker).

Ideally it is harmless as it is retrial, It can retry and try to send request to other brokers

Reference Link:http://kafka.apache.org/documentation/#replication

On what basis does kafka leader election happen ?

Kafka maintains a set of in-sync replicas. Only the members of this set are eligible for election as leader. It will be stored in ZK.

Sharing quote from Kafka document “Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster”

Reference Link: https://kafka.apache.org/documentation#design_replicatedlog

What are all the reasons that cause kafka partition leader election to trigger?

1. When the active leader crashed or failed, New leader will be selected from one of the in-sync replicas
2. If you have enabled “auto.leader.rebalance.enable=true” and if your preferred leader is not leader. It will try to make it as leader when it is in in-sync replica.

Hope it helps, Let us know if you any questions

Thanks

Jerry