Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka Broker new consumer with multiple NICs doesnt work WITHOUT zookeeper

Highlighted

Kafka Broker new consumer with multiple NICs doesnt work WITHOUT zookeeper

New Contributor

Hi.

This is my setup:

  • 3 kafka hosts with 2 networks
  • private internal network for data
  • public for connection with everything else
+--------------------------------------------------------------------------------------------------+        
|                                                                                                  |        |                                10.25.12.0/24 - PUBLIC                                            |        
|                                                                                                  |        
|         ---------------------------------------------------------------                          |     
|                |                        |                        |                               |        
|                |                        |                        |                               |        
|                |                        |                        |                               |        
|          +------------+           +------------+          +------------+                         |        
|          |            |           |            |          |            |                         |        
|          |            |           |            |          |            |                         |        
|          |    KFK-1   |           |    KFK-2   |          |    KFK-3   |                         |        
|          |            |           |            |          |            |                         |        
|          |            |           |            |          |            |                         |        
|          +------------+           +------------+          +------------+                         |       
|                |                        |                       |                                |        
|                -                        |                       |                                |        
|                |                        |                       |                                |       
|                |                        |                       |                                |        
|          ---------------------------------------------------------------                         |        
|                             192.168.160.0/24 - PRIVATE                                           |        
|          server.properties                                                                       |       
|          advertised.listeners=PLAINTEXT://10.25.12.[1,2,3]:6667                                  |        
|          listeners=PLAINTEXT://0.0.0.0:6667                                                      |        
|          inter.broker.listener.name=PLAINTEXT://192.168.160.[1,2,3]                              |        
|          zookeeper.connect=kfk-1.domain.tld:2181,kfk-2.domain.tld:2181,kfk-3.domain.tld:2181     |        
|                                                                                                  |        
+--------------------------------------------------------------------------------------------------+        

Every host, to use the data network is routed through the hosts file to the internal one:

#INTERNAL NETWORK 

192.168.160.1		kfk-1.domain.tld
192.168.160.2         	kfk-2.domain.tld
192.168.160.3		kfk-3.domain.tld

The problem:

If i try to connect from the outside to a round-robin hostname which contains all the 3 public private addresses of the kafka-brokes using zookeeper (kafka.domain.tld:2181) > works

If i try to connect to ANY of the brokers directly using bootstrat-server > doesnt even throw me and error.

I tried placing on the advertised.listeners:

  1. plaintext://kfk-[1,2,3].domain.tld:6667 >> connect but can not find the leader
  2. plaintext://0.0.0.0:6667 >> dont connect
  3. plaintext://kafka.domain.tld:6667 >> dont connect

Running HDP 2.6.1.

I am ran out of ideas.....any clue?

Don't have an account?
Coming from Hortonworks? Activate your account here