Kafka console producer/consumer failing with AD users but works with local kafka realm.

[#1]

Hi,

We have just built a new kafka cluster(3.1.0-1.3.1.0.p0.35) and integrated it with kerberos. Kerberos is integrated with AD. We are able to produce and consume with kafka principle which is local but with AD users, it fails with below error on console.

19/06/09 08:14:22 DEBUG authenticator.SaslClientAuthenticator: Set SASL client state to CLIENT_COMPLETE
19/06/09 08:14:22 DEBUG authenticator.SaslClientAuthenticator: Set SASL client state to COMPLETE
19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Initiating API versions fetch from node -1.
19/06/09 08:14:22 DEBUG network.Selector: [Producer clientId=console-producer] Connection with server2.kafka4.corp/180.20.92.23 disconnected
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:545)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:483)
at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:481)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
at java.lang.Thread.run(Thread.java:748)
19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Node -1 disconnected.
19/06/09 08:14:22 INFO clients.NetworkClient: [Producer clientId=console-producer] API versions request failed via disconnect. Defaulting legacy API versions
19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Give up sending metadata request since no node is available
19/06/09 08:14:22 DEBUG clients.NetworkClient: [Producer clientId=console-producer] Give up sending metadata request since no node is available
^C19/06/09 08:14:22 INFO producer.KafkaProducer: [Producer clientId=console-producer] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
19/06/09 08:14:22 DEBUG internals.Sender: [Producer clientId=console-producer] Beginning shutdown of Kafka producer I/O thread, sending remaining records.

==========================================

The broker logs only have errors about the shortname for users.

Caused by: org.apache.kafka.common.security.kerberos.NoMatchingRule: No rules apply to RAJESH@KAFKA4.CORP, rules [DEFAULT]

[#2]

Hello @RajeshBodolla 

It seems like the actual exception is as below

Caused by: org.apache.kafka.common.security.kerberos.NoMatchingRule: No rules apply to RAJESH@KAFKA4.CORP, rules [DEFAULT]

Could you please share the complete stack? it seems it is somewhere not able to resolve the principal

[#3 · Accepted solution]

@satz 

We were able to resolve this. We had the kerberos auth principles in default kafka group while all the broker were in a different config group. Adding the auth principles to the kafka config group has solved the issue.