Support Questions

Find answers, ask questions, and share your expertise

In Kafak, what is the difference while producing with single broker host and multiple broker host ?

Explorer

In Kafak, what is the difference while producing with single broker host and multiple broker host ? For example, Single Broker Hosts

# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list hadoop1.test.com:6667 --topic testtopic --security-protocol PLAINTEXTSASL
For example, Multiple Broker Hosts
# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list hadoop1.test.com:6667,hadoop2.test.com:6667 --topic testtopic --security-protocol PLAINTEXTSASL
1 ACCEPTED SOLUTION

Hi @Tech Gig

Let's say you have a cluster of 3 brokers. In case 1 (--broker-list hadoop1.test.com:6667) a client that connects for the first time to the cluster try to reach the broker you specified. If this broker is not reachable (host is down, network issue, etc) the connection fails. The connection fails even if there are other brokers that can do the job and provide the required information.

When you provide a list of brokers, the client uses each broker sequentially until first successful response. You have a better HA in this case.

View solution in original post

4 REPLIES 4

Hi @Tech Gig

Let's say you have a cluster of 3 brokers. In case 1 (--broker-list hadoop1.test.com:6667) a client that connects for the first time to the cluster try to reach the broker you specified. If this broker is not reachable (host is down, network issue, etc) the connection fails. The connection fails even if there are other brokers that can do the job and provide the required information.

When you provide a list of brokers, the client uses each broker sequentially until first successful response. You have a better HA in this case.

Explorer

Thank you for the simple and crisp explanation

In a Kafka cluster, there are multiple brokers to support the design goals of availability and high throughput.

The broker list defines where the Producer can find a one or more Brokers to determine the Leader for each topic. This does not need to be the full set of Brokers in your cluster, but should include at least two in case the first Broker is not available.

One needn't worry about figuring out which Broker is the leader for the topic (and partition), as the Producer knows how to connect to the Broker and ask for the metadata and then connect to the correct Broker.

Explorer

Thanks slachterman

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.