Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

In Kafak, what is the difference while producing with single broker host and multiple broker host ?

avatar
Contributor

In Kafak, what is the difference while producing with single broker host and multiple broker host ? For example, Single Broker Hosts

# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list hadoop1.test.com:6667 --topic testtopic --security-protocol PLAINTEXTSASL
For example, Multiple Broker Hosts
# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list hadoop1.test.com:6667,hadoop2.test.com:6667 --topic testtopic --security-protocol PLAINTEXTSASL
1 ACCEPTED SOLUTION

avatar

Hi @Tech Gig

Let's say you have a cluster of 3 brokers. In case 1 (--broker-list hadoop1.test.com:6667) a client that connects for the first time to the cluster try to reach the broker you specified. If this broker is not reachable (host is down, network issue, etc) the connection fails. The connection fails even if there are other brokers that can do the job and provide the required information.

When you provide a list of brokers, the client uses each broker sequentially until first successful response. You have a better HA in this case.

View solution in original post

4 REPLIES 4

avatar

Hi @Tech Gig

Let's say you have a cluster of 3 brokers. In case 1 (--broker-list hadoop1.test.com:6667) a client that connects for the first time to the cluster try to reach the broker you specified. If this broker is not reachable (host is down, network issue, etc) the connection fails. The connection fails even if there are other brokers that can do the job and provide the required information.

When you provide a list of brokers, the client uses each broker sequentially until first successful response. You have a better HA in this case.

avatar
Contributor

Thank you for the simple and crisp explanation

avatar

In a Kafka cluster, there are multiple brokers to support the design goals of availability and high throughput.

The broker list defines where the Producer can find a one or more Brokers to determine the Leader for each topic. This does not need to be the full set of Brokers in your cluster, but should include at least two in case the first Broker is not available.

One needn't worry about figuring out which Broker is the leader for the topic (and partition), as the Producer knows how to connect to the Broker and ask for the metadata and then connect to the correct Broker.

avatar
Contributor

Thanks slachterman