how to configure Kafka so it's listening on two networks

[#1]

I have an HDP cluster that's running two networks: a 10.x network for external traffic and a 192.x network for internal traffic. Kafka is listening on the the 192.x network, and I have a client application running that needs to write to Kafka that's running on the 10.x network. I need Kafka to listen on both networks.

The `listeners` property is currently set to `PLAINTEXT://localhost:6667`. I stumbled across a blog that says we can replace localhost with an actual IP address, e.g. `PLAINTEXT://10.x.x.x:6667,PLAINTEXT://192.x.x.x:6667`. If I set this property in Ambari, this property gets pushed to all the brokers. Obviously, the IP addresses will differ from host to host.

Is there a way for Ambari to have host-specific properties? Or is there another way to configure Kafka, via Ambari, where it's listening on multiple NICs?

[#2]

@Alex Woolford

Look at:

• https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+tra...
• https://issues.apache.org/jira/browse/KAFKA-4565

If helpful, please vote and accept best answer.

[#3 · Accepted solution]

The solution to this was to create a config group for each broker and add a host-specific `advertised.listeners` property in the Custom kafka-broker section.

This has been beautifully documented in this YouTube video: https://www.youtube.com/watch?v=bRKw5y_tx5E

[#4]

Hi,

Instead of creating separate CGROUP for each Broker node in Kafka cluster, we can use kafka env to make it working.

• To configure Kafka to advertise FQDN and listening on all the IP addresses, add the following text to the bottom of the kafka-env-template.

# Configure Kafka to advertise IP addresses instead of FQDN
HOST_FQDN=$(hostname -f)
echo advertised.listeners=$HOST_FQDN
sed -i.bak -e '/advertised/{/advertised@/!d;}' /usr/hdp/current/kafka-broker/conf/server.properties
echo "advertised.listeners=SASL_PLAINTEXT:://$HOST_FQDN:6667" >> /usr/hdp/current/kafka-broker/conf/server.properties

• To configure Kafka to listen on all network interfaces, change the value in the listeners field to SASL_PLAINTEXT:PLAINTEXT://0.0.0.0:6667

Reference from : https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-connect-vpn-gateway

Thanks,

Saravana