Any ideas on consuming a Kerberos secured Kafka topic from a .NET client?
So we got a Kafka setup on Linux (Cloudera Kafka v2.0) configured with Kerberos and everything is working fine with Java clients using a keytab file.
Now we would like to enable communication from .NET clients to the Kafka broker, and that too works fine as long as we are using unsecured topics, but company policies dictate that we use Kerberos here too, and that’s where the problems arise.
The C# .NET client we are using is this one: https://github.com/Jroland/kafka-net/pull/84
We have wrapped the NetworkStream in a NegotiateStream as suggested, and below is the section where we try to authenticate with Kerberos.
// Setup technical user to access Kafka
NegotiateStream authStream = new NegotiateStream(stream, false);
NetworkCredential nc = new NetworkCredential("<user>", "<password>", "<domain>");
// Pass the NegotiateStream as the AsyncState object
// so that it is available to the callback delegate.
IAsyncResult ar = authStream.BeginAuthenticateAsClient(nc, "<SPN>",
First we specify a technical user we have on the domain.
Next we call “BeginAuthenticateAsClient” referring the technical user, and the SPN (Service Principle Name) of Kafka, but the callback is never reached.
Actually the SPN is likely to be the problem, as we are not sure exactly what to specify, or whether something should be registered in the AD for the SPN to be “officially” recognised.
This is likely the case and the AD (KDC) probably knows nothing about the Linux brokers.
Microsoft articles state that the SPN should have a format like : <service class>/<host>:<port>/<service name>
That would resolve to something like kafka/<broker> but that is not what the Java Kafka client use. Their “Principal” is set as <user>@<realm>.
As mentioned, our brokers are running on Linux, so is it possible at all to register an SPN making the brokers official on a Windows AD (KDC), or are those worlds not compatible?
Any comments or hints would be highly appreciated.
BTW : Im from the Windows side, so what I wrote aboute "the other side" isn't neccesarily correct :O)
I am in the same situation. Were you able to solve this problem? I am also trying to connect my dotnet producer client to linux brokers using Kerberos.
Our team still has this issue too, as far as I know, there still isn't a way. But we have worked around for the time being by opening a non-Kerberized port for Kafka (obviously a bad solution). There are talks of incorporating AD in our Linux environment, which should make this a non-issue for us.
I don't see a description of what the problem is. Based on what I read of the "BeginAuthenticateAsClient" the SPN will be what is to the left of the "@" sign in the principal for the Kafaka server's principal.
That said, if authenticated is attempted, I suggest making sure the broker received the authentication request. Logging on the broker side may be helpful, debugging in the .NET code, or wireshark packet capture may help.
You may also want to consider using a synchronous authentication call with AuthenticateAsClient as that may show more of what the problem is. Assuming there should be a success or failure response, the code should then block till it gets that. Not sure if that will help, but it may be that the async call is not allowing you to see the error or problem.
Also, make sure that .NET is able to get a service ticket to authenticate via kerberos to the broker. I've seen browsers hang while waiting for SSPI to return if there is an underlying kerberos problem.
Just some ideas on how to approach this issue... I've never tried it myself.