In Kafak, what is the difference while producing with single broker host and multiple broker host ?

[#1]

In Kafak, what is the difference while producing with single broker host and multiple broker host ? For example, Single Broker Hosts

# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list hadoop1.test.com:6667 --topic testtopic --security-protocol PLAINTEXTSASL

For example, Multiple Broker Hosts

# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list hadoop1.test.com:6667,hadoop2.test.com:6667 --topic testtopic --security-protocol PLAINTEXTSASL

[#2 · Accepted solution]

Hi @Tech Gig

Let's say you have a cluster of 3 brokers. In case 1 (--broker-list hadoop1.test.com:6667) a client that connects for the first time to the cluster try to reach the broker you specified. If this broker is not reachable (host is down, network issue, etc) the connection fails. The connection fails even if there are other brokers that can do the job and provide the required information.

When you provide a list of brokers, the client uses each broker sequentially until first successful response. You have a better HA in this case.

[#3]

Thank you for the simple and crisp explanation

[#4]

In a Kafka cluster, there are multiple brokers to support the design goals of availability and high throughput.

The broker list defines where the Producer can find a one or more Brokers to determine the Leader for each topic. This does not need to be the full set of Brokers in your cluster, but should include at least two in case the first Broker is not available.

One needn't worry about figuring out which Broker is the leader for the topic (and partition), as the Producer knows how to connect to the Broker and ask for the metadata and then connect to the correct Broker.

[#5]

Thanks slachterman