Kafka behind an external Load Balancer

[#1]

Hi,

We need to create data flow between two kafka clusters (cluster1 and cluster2). We are using Nifi sitting at cluster1 to send the messages from cluster1 kafka to cluster2 kafka.

Here we wanted to use the external load balancer in front of cluster2 Kafka. However, it is not working properly. The Nifi is unable to send the data to cluster2 kafka if we use LB url in the kafka brokers section. If we put the actual kafka brokers, it is working properly. Can any one help me on this asap.

[#2]

Is there any specific reason why you need a load balancer? Kafka is supposed to work without a separated load balancer, and handle the load on the cluster. With the load balancer the client might fail to connect when the broker tries to redirect the connections to another listener.

The brokers section in the config for a client is used to try getting the actual connect parameter (listeners) from the cluster. You typically provide multiple brokers to avoid having an issue if the initial broker is down while connecting.

When you try to change the listener configs in the kafka brokers (or zookeeper), i think you actually disable the cluster, as the broker are communicating to each other as well.

[#3]

Load balancers would help in the case where you want a more friendly name than some DNS records or the case where IP's are dynamic.

Besides that, remembering one address is easier than a long list of 3-5 servers.

[#4]

Is there any way to do so ?