question Re: how to configure Kafka so it's listening on two networks in Archives of Support Questions (Read Only)

how to configure Kafka so it's listening on two networks

alexw — Thu, 29 Mar 2018 21:20:32 GMT

I have an HDP cluster that's running two networks: a 10.x network for external traffic and a 192.x network for internal traffic. Kafka is listening on the the 192.x network, and I have a client application running that needs to write to Kafka that's running on the 10.x network. I need Kafka to listen on both networks.

The `listeners` property is currently set to `PLAINTEXT://localhost:6667`. I stumbled across a blog that says we can replace localhost with an actual IP address, e.g. `PLAINTEXT://10.x.x.x:6667,PLAINTEXT://192.x.x.x:6667`. If I set this property in Ambari, this property gets pushed to all the brokers. Obviously, the IP addresses will differ from host to host.

Is there a way for Ambari to have host-specific properties? Or is there another way to configure Kafka, via Ambari, where it's listening on multiple NICs?

Re: how to configure Kafka so it's listening on two networks

cstanca — Sat, 31 Mar 2018 01:24:29 GMT

@Alex Woolford

Look at:

If helpful, please vote and accept best answer.

Re: how to configure Kafka so it's listening on two networks

alexw — Thu, 03 May 2018 14:06:43 GMT

The solution to this was to create a config group for each broker and add a host-specific `advertised.listeners` property in the Custom kafka-broker section.

This has been beautifully documented in this YouTube video: https://www.youtube.com/watch?v=bRKw5y_tx5E

Re: how to configure Kafka so it's listening on two networks

saravanakumar_n — Fri, 30 Aug 2019 18:19:41 GMT

Hi,

Instead of creating separate CGROUP for each Broker node in Kafka cluster, we can use kafka env to make it working.

To configure Kafka to advertise FQDN and listening on all the IP addresses, add the following text to the bottom of the kafka-env-template.

# Configure Kafka to advertise IP addresses instead of FQDN
HOST_FQDN=$(hostname -f)
echo advertised.listeners=$HOST_FQDN
sed -i.bak -e '/advertised/{/advertised@/!d;}' /usr/hdp/current/kafka-broker/conf/server.properties
echo "advertised.listeners=SASL_PLAINTEXT:://$HOST_FQDN:6667" >> /usr/hdp/current/kafka-broker/conf/server.properties

To configure Kafka to listen on all network interfaces, change the value in the listeners field to SASL_PLAINTEXT:PLAINTEXT://0.0.0.0:6667

Reference from : https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-connect-vpn-gateway

Thanks,

Saravana