Reply
Explorer
Posts: 41
Registered: ‎03-25-2017

Kafka failing to connect kerberized kafka broker using nginx

Hi,

 

I have a usecase where i want my producers/consumers to connect to my kerberized kafka cluster/broker only through nginx server to hide the broker IP and also to perform load balancing. I have setup  kerberos kafka cluster and able to consume and produce data without nginx. But when i try to consume/produce the data from/to kafka,it fails with timeout error.After enabling DEBUG in kakfa server.log i can see  nginx server ip but  couldnot find any error which could help me to find out the cause of failure.

 

This 

 

Warm regards

Sidharth

Explorer
Posts: 41
Registered: ‎03-25-2017

Re: Kafka failing to connect kerberized kafka broker using nginx

Can anyone please reply
New Contributor
Posts: 2
Registered: ‎02-19-2018

Re: Kafka failing to connect kerberized kafka broker using nginx

What client versions are you using? We had run into something similar when connecting the Striim server to kerberized kafka. Ultimately, in our case we were timing out when fetching the topic metadata.  This was the setting that worked... 
metadata.fetch.timeout.ms=300000;

Explorer
Posts: 41
Registered: ‎03-25-2017

Re: Kafka failing to connect kerberized kafka broker using nginx

Thank you for your help. For me it's like I am connecting to the same server . For example- I am running Kafka producer from server1.abc.com to broker running on same node server1.abc.com without any proxy work fine and can produce but if the same I try with proxy doesn't work and goes timeout but I can see proxy server entry at targeted broker log and also in tcpdump
Explorer
Posts: 41
Registered: ‎03-25-2017

Re: Kafka failing to connect kerberized kafka broker using nginx

Also the provided configuration is deprecated. So could you please help me with the working one for latest stable version.
New Contributor
Posts: 2
Registered: ‎02-19-2018

Re: Kafka failing to connect kerberized kafka broker using nginx

Ok. I understand. Indeed the config has been deprecated, that's why I
wanted to know the version.
This sounds like more of a config setting using a proxy.
The following link may help you. I'll also update once I have a chance to
try It out.

https://stackoverflow.com/questions/45297528/how-can-we-configure-kafka-producer-behind-a-firewall-p...
Explorer
Posts: 41
Registered: ‎03-25-2017

Re: Kafka failing to connect kerberized kafka broker using nginx

Thank you!
I have two Kafka cluster now with one kerberos and another without
kerberos

Tried 3 approach

Approach 1-

Produce and consume data with Kerberos Kafka and without nginx proxy, work
fine and able to consume and publish data in Kafka topics.

Approach 2-

Produce and consume data with Kerberos Kafka and with nginx proxy, doesn't
work and get timeout error in producer and consumer unable to get data
from Kafka topics

Approach 3-

Produce and consume data without kerberos and with/without nginx proxy,
work fine and able to consume and publish data from/to Kafka topics.


For security concern we can't proceed no kerberos setup and stuck with
nginx issue. Please help my Kafka version is 0.10.1.

Cloudera Employee
Posts: 232
Registered: ‎01-09-2014

Re: Kafka failing to connect kerberized kafka broker using nginx

The reason that you can't produce when fronted by nginx is that kerberos needs the service principal name to match the name of the host that you are connecting to. Using a load balancer for any kerberized service requires you to have keytabs that match the hostname, as well as the load balancer dns name. Currently this functionality is not built into CDH kafka.

You could try a 1 to 1 mapping on your nginx server using advertised.listener property, but there may be some integration issues that you have to work through when you are doing that.

The issue is that kafka advertises the hostnames in the metadata that the client will need to connect to. Those hostnames must be resolvable and have proper keytab entries for both the client as well as inter broker communication.
-pd
Explorer
Posts: 41
Registered: ‎03-25-2017

Re: Kafka failing to connect kerberized kafka broker using nginx

Thanks for your response. Got much better understanding now. However, I
would like to say I have created a principal something like "
centos@CLOUDERA.COM" and the same is working when I am trying to connect
kerberized broker directly without nginx. But the same doesn't work with
nginx.


Thanks

Sidharth
Cloudera Employee
Posts: 232
Registered: ‎01-09-2014

Re: Kafka failing to connect kerberized kafka broker using nginx

So, the additional issue you are facing is that when you receive the metadata from the broker cluster, it notes which brokers host specific partitions, and uses the advertised.listeners property as the hostname of the broker machine so the client knows which hosts to connect to. The client doesn't proxy all connections through a single host/load balancer, it needs to connect to each individual host. You can handle that with the load balancer if you create a 1 to 1 mapping of VIPs to hostnames, and then set the advertised.listeners property to that mapping, uniquely for each host. You would then also need to set inter.broker.listener.name properly [1] so that the brokers won't use that VIP address to communicate with eachother, but will use internal addresses.
[1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+tra...

This assumes that you are running CDK 2.2 or 3.0 where KIP-103 is included.

-pd
Announcements