- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
how to configure Kafka so it's listening on two networks
- Labels:
-
Apache Ambari
-
Apache Kafka
Created 03-29-2018 02:20 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have an HDP cluster that's running two networks: a 10.x network for external traffic and a 192.x network for internal traffic. Kafka is listening on the the 192.x network, and I have a client application running that needs to write to Kafka that's running on the 10.x network. I need Kafka to listen on both networks.
The `listeners` property is currently set to `PLAINTEXT://localhost:6667`. I stumbled across a blog that says we can replace localhost with an actual IP address, e.g. `PLAINTEXT://10.x.x.x:6667,PLAINTEXT://192.x.x.x:6667`. If I set this property in Ambari, this property gets pushed to all the brokers. Obviously, the IP addresses will differ from host to host.
Is there a way for Ambari to have host-specific properties? Or is there another way to configure Kafka, via Ambari, where it's listening on multiple NICs?
Created 05-03-2018 07:06 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The solution to this was to create a config group for each broker and add a host-specific `advertised.listeners` property in the Custom kafka-broker section.
This has been beautifully documented in this YouTube video: https://www.youtube.com/watch?v=bRKw5y_tx5E
Created 03-30-2018 06:24 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Look at:
- https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+tra...
- https://issues.apache.org/jira/browse/KAFKA-4565
If helpful, please vote and accept best answer.
Created 05-03-2018 07:06 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The solution to this was to create a config group for each broker and add a host-specific `advertised.listeners` property in the Custom kafka-broker section.
This has been beautifully documented in this YouTube video: https://www.youtube.com/watch?v=bRKw5y_tx5E
Created 08-30-2019 11:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Instead of creating separate CGROUP for each Broker node in Kafka cluster, we can use kafka env to make it working.
- To configure Kafka to advertise FQDN and listening on all the IP addresses, add the following text to the bottom of the kafka-env-template.
# Configure Kafka to advertise IP addresses instead of FQDN
HOST_FQDN=$(hostname -f)
echo advertised.listeners=$HOST_FQDN
sed -i.bak -e '/advertised/{/advertised@/!d;}' /usr/hdp/current/kafka-broker/conf/server.properties
echo "advertised.listeners=SASL_PLAINTEXT:://$HOST_FQDN:6667" >> /usr/hdp/current/kafka-broker/conf/server.properties
- To configure Kafka to listen on all network interfaces, change the value in the listeners field to SASL_PLAINTEXT:PLAINTEXT://0.0.0.0:6667
Reference from : https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-connect-vpn-gateway
Thanks,
Saravana
