Support Questions
Find answers, ask questions, and share your expertise

how to configure Kafka so it's listening on two networks

New Contributor

I have an HDP cluster that's running two networks: a 10.x network for external traffic and a 192.x network for internal traffic. Kafka is listening on the the 192.x network, and I have a client application running that needs to write to Kafka that's running on the 10.x network. I need Kafka to listen on both networks.

The `listeners` property is currently set to `PLAINTEXT://localhost:6667`. I stumbled across a blog that says we can replace localhost with an actual IP address, e.g. `PLAINTEXT://10.x.x.x:6667,PLAINTEXT://192.x.x.x:6667`. If I set this property in Ambari, this property gets pushed to all the brokers. Obviously, the IP addresses will differ from host to host.

Is there a way for Ambari to have host-specific properties? Or is there another way to configure Kafka, via Ambari, where it's listening on multiple NICs?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to configure Kafka so it's listening on two networks

New Contributor

The solution to this was to create a config group for each broker and add a host-specific `advertised.listeners` property in the Custom kafka-broker section.

This has been beautifully documented in this YouTube video: https://www.youtube.com/watch?v=bRKw5y_tx5E

View solution in original post

3 REPLIES 3

Re: how to configure Kafka so it's listening on two networks

Re: how to configure Kafka so it's listening on two networks

New Contributor

The solution to this was to create a config group for each broker and add a host-specific `advertised.listeners` property in the Custom kafka-broker section.

This has been beautifully documented in this YouTube video: https://www.youtube.com/watch?v=bRKw5y_tx5E

View solution in original post

Re: how to configure Kafka so it's listening on two networks

New Contributor

Hi,

 

Instead of creating separate CGROUP for each Broker node in Kafka cluster, we can use kafka env to make it working.

 

  • To configure Kafka to advertise FQDN and listening on all the IP addresses, add the following text to the bottom of the kafka-env-template.

# Configure Kafka to advertise IP addresses instead of FQDN
HOST_FQDN=$(hostname -f)
echo advertised.listeners=$HOST_FQDN
sed -i.bak -e '/advertised/{/advertised@/!d;}' /usr/hdp/current/kafka-broker/conf/server.properties
echo "advertised.listeners=SASL_PLAINTEXT:://$HOST_FQDN:6667" >> /usr/hdp/current/kafka-broker/conf/server.properties

 

  • To configure Kafka to listen on all network interfaces, change the value in the listeners field to SASL_PLAINTEXT:PLAINTEXT://0.0.0.0:6667

Reference from : https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-connect-vpn-gateway

Thanks,

Saravana