Kafka leader election

[#1]

Hi All,

Curious to know about , how kafka works internally

val producer = new KafkaProducer(“test-topic”,"192.168.86.10:9092")

producer.send(“hello distributed commit log”);

From the above code, Producer is sending message to the kafka cluster.

What will happen if that node fails ? Is this single point of failure? if NO, how the new leader is elected ? what are the creteria?

[#2 · Accepted solution]

Hello!

This is definitely not a single point of failure in a Kafka cluster. Let me quote from the Kafka doc:

"Each partition has one server which acts as the "leader" and zero or more servers which act as "followers". The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster."

And there is a nice explanation of how leader election is performed:

https://kafka.apache.org/documentation#design_replicatedlog

"Kafka takes a slightly different approach to choosing its quorum set. Instead of majority vote, Kafka dynamically maintains a set of in-sync replicas (ISR) that are caught-up to the leader. Only members of this set are eligible for election as leader. A write to a Kafka partition is not considered committed until all in-sync replicas have received the write. This ISR set is persisted to ZooKeeper whenever it changes. Because of this, any replica in the ISR is eligible to be elected leader."

[#3]

Thanks for your quick reply.one quick question, I am sending message to particular IP and this IP belongs to one physical machine, when this machine went down, then IP is not reachble ( though kafka cluster elects new leader , its IP may be different..) . in this case it is sinle point of faliure isn't it ?

[#4]

Hi Gobi,

In your KafkaProducer constructor, you instantiate the class with a set of Properties, which should include a list of Brokers. This allows the Producer to have knowledge of more than one server. If you only have one server listed, then, yes, if that server goes down, your Producer will be unable to send any more messages. However, this scenario is highly unlikely because it is a best practice to use more than one Broker in your cluster. One benefit of configuring your Producer with a list of servers allows you to send messages without having to worry about the IP address of the particular server that will receive your messages.

In terms of defining the topic, to which you will send your messages, this is defined in the ProducerRecord and can be achieved with something like this:

Properties props = new Properties();
props.put(“bootstrap.servers”, “192.168.86.10:9092, host2:port, host3:port”);
Producer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<String, String>(“test-topic”, “hello distributed commit log”));

Have a great day,

Brian