Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to configure Kafka so it's listening on two networks

Solved Go to solution
Highlighted

how to configure Kafka so it's listening on two networks

New Contributor

I have an HDP cluster that's running two networks: a 10.x network for external traffic and a 192.x network for internal traffic. Kafka is listening on the the 192.x network, and I have a client application running that needs to write to Kafka that's running on the 10.x network. I need Kafka to listen on both networks.

The `listeners` property is currently set to `PLAINTEXT://localhost:6667`. I stumbled across a blog that says we can replace localhost with an actual IP address, e.g. `PLAINTEXT://10.x.x.x:6667,PLAINTEXT://192.x.x.x:6667`. If I set this property in Ambari, this property gets pushed to all the brokers. Obviously, the IP addresses will differ from host to host.

Is there a way for Ambari to have host-specific properties? Or is there another way to configure Kafka, via Ambari, where it's listening on multiple NICs?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: how to configure Kafka so it's listening on two networks

New Contributor

The solution to this was to create a config group for each broker and add a host-specific `advertised.listeners` property in the Custom kafka-broker section.

This has been beautifully documented in this YouTube video: https://www.youtube.com/watch?v=bRKw5y_tx5E

View solution in original post

3 REPLIES 3

Re: how to configure Kafka so it's listening on two networks

Highlighted

Re: how to configure Kafka so it's listening on two networks

New Contributor

The solution to this was to create a config group for each broker and add a host-specific `advertised.listeners` property in the Custom kafka-broker section.

This has been beautifully documented in this YouTube video: https://www.youtube.com/watch?v=bRKw5y_tx5E

View solution in original post

Highlighted

Re: how to configure Kafka so it's listening on two networks

New Contributor

Hi,

 

Instead of creating separate CGROUP for each Broker node in Kafka cluster, we can use kafka env to make it working.

 

  • To configure Kafka to advertise FQDN and listening on all the IP addresses, add the following text to the bottom of the kafka-env-template.

# Configure Kafka to advertise IP addresses instead of FQDN
HOST_FQDN=$(hostname -f)
echo advertised.listeners=$HOST_FQDN
sed -i.bak -e '/advertised/{/advertised@/!d;}' /usr/hdp/current/kafka-broker/conf/server.properties
echo "advertised.listeners=SASL_PLAINTEXT:://$HOST_FQDN:6667" >> /usr/hdp/current/kafka-broker/conf/server.properties

 

  • To configure Kafka to listen on all network interfaces, change the value in the listeners field to SASL_PLAINTEXT:PLAINTEXT://0.0.0.0:6667

Reference from : https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-connect-vpn-gateway

Thanks,

Saravana

 

Don't have an account?
Coming from Hortonworks? Activate your account here