Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to set kafka to broadcast the ip address rather than host name?

Highlighted

How to set kafka to broadcast the ip address rather than host name?

Rising Star

I use kafka with default value(PLAINTEXT://localhost:9092), however, it broadcasts it's hostname instead of ip address. The result is, if the producer runs a host without the kafka host info, it cannot send messages to kafka. How to resolve it?

1 REPLY 1

Re: How to set kafka to broadcast the ip address rather than host name?

New Contributor

We ran into the same issue because we rely on poor mans DNS via local hosts file, as we don't have control over the infrastructure.

To solve this issue of advertising non existent hostnames there are two solutions:
1. create separate configuration groups for each kafka broker and override `listeners` property with explicit IP of the relevant node
2. setup ambari-agent to specify a custom public hostname and use template variable at kafka config to use that property

Solution 2 introduces a fix or another problem, it depends on your setup: By setting up ambari-agent to use a custom public hostname links from ambari to services like HDFS UI, YARN UI, Spark UI, Zeppelin, etc. will use this value.

To setup solution 2:

Create public hostname script

Place a file at /var/lib/ambari-agent/public_hostname.sh and make it executable chmod a+x /var/lib/ambari-agent/public_hostname.sh with following content:

#!/bin/sh
hostname -I | awk '{print $1}'
Change ambari-agent config

at /etc/ambari-agent/conf/ambari-agent.ini, add property at agent section:

public_hostname_script=/var/lib/ambari-agent/public_hostname.sh
Restart ambari agent
ambari-agent restart
Configure Kafka Broker listener
PLAINTEXT://{{config['agentLevelParams']['public_hostname']}}:6667
Restart Kafka
Don't have an account?
Coming from Hortonworks? Activate your account here