Welcome to the Cloudera Community
Your Enterprise Data Cloud Community. With over 87,300 members and 19,900 solutions, you've come to the right place!

Who agreed with this topic

Communicating from .NET to Kafka using Kerberos

New Contributor

Any ideas on consuming a Kerberos secured Kafka topic from a .NET client?

 

Background:

So we got a Kafka setup on Linux (Cloudera Kafka v2.0) configured with Kerberos and everything is working fine with Java clients using a keytab file.

 

Now we would like to enable communication from .NET clients to the Kafka broker, and that too works fine as long as we are using unsecured topics, but company policies dictate that we use Kerberos here too, and that’s where the problems arise.

 

The C# .NET client we are using is this one: https://github.com/Jroland/kafka-net/pull/84

 

We have wrapped the NetworkStream in a NegotiateStream as suggested, and below is the section where we try to authenticate with Kerberos.

 

           // Setup technical user to access Kafka

           NegotiateStream authStream = new NegotiateStream(stream, false);

           NetworkCredential nc = new NetworkCredential("<user>", "<password>", "<domain>");

 

           // Pass the NegotiateStream as the AsyncState object

           // so that it is available to the callback delegate.

           IAsyncResult ar = authStream.BeginAuthenticateAsClient(nc, "<SPN>",

               ProtectionLevel.EncryptAndSign,

               TokenImpersonationLevel.Impersonation,

               new AsyncCallback(EndAuthenticateCallback),

               authStream

               );

 

First we specify a technical user we have on the domain.

 

Next we call “BeginAuthenticateAsClient” referring the technical user, and the SPN (Service Principle Name) of Kafka, but the callback is never reached.

 

Actually the SPN is likely to be the problem, as we are not sure exactly what to specify, or whether something should be registered in the AD for the SPN to be “officially” recognised.

This is likely the case and the AD (KDC) probably knows nothing about the Linux brokers.

 

Microsoft articles state that the SPN should have a format like : <service class>/<host>:<port>/<service name>

 

That would resolve to something like kafka/<broker> but that is not what the Java Kafka client use. Their “Principal” is set as <user>@<realm>.

 

As mentioned, our brokers are running on Linux, so is it possible at all to register an SPN making the brokers official on a Windows AD (KDC), or are those worlds not compatible?

 

Any comments or hints would be highly appreciated.

 

BTW : Im from the Windows side, so what I wrote aboute "the other side" isn't neccesarily correct :O)

Who agreed with this topic