Kafka broker SASL connection failing with Zookeeper

[#1]

Dear experts,

I have installed apache kafka 2.4 on one node. I am having SSL and SASL(Kerberos) enabled for kafka broker and now enabled SASL for zookeeper. However when starting the broker , i am getting the below error, could you please help on this ?

--error log---

[2020-09-07 14:11:09,761] DEBUG Client principal is "kafka/broker0@KAFKA.SECURE". (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,761] DEBUG Server principal is "krbtgt/KAFKA.SECURE@KAFKA.SECURE". (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,764] INFO TGT valid starting at: Mon Sep 07 14:11:09 IST 2020 (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,765] INFO TGT expires: Tue Sep 08 14:11:09 IST 2020 (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,765] INFO TGT refresh sleeping until: Tue Sep 08 09:30:58 IST 2020 (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,765] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-09-07 14:11:09,766] DEBUG creating sasl client: Client=kafka/broker0@KAFKA.SECURE;service=kafka;serviceHostname=broker0 (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-09-07 14:11:09,773] INFO Opening socket connection to server broker0/X.X.X.X:2181. Will attempt to SASL-authenticate using Login Context section 'ZkClient' (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,778] INFO Socket connection established, initiating session, client: /X.X.X.X:54728, server: broker0/X.X.X.X:2181 (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,780] DEBUG Session establishment request sent on broker0/X.X.X.X:2181 (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,785] INFO Session establishment complete on server broker0/X.X.X.X:2181, sessionid = 0x100000039900003, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,786] DEBUG ClientCnxn:sendSaslPacket:length=0 (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-09-07 14:11:09,787] DEBUG saslClient.evaluateChallenge(len=0) (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-09-07 14:11:09,789] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2020-09-07 14:11:09,811] ERROR SASL authentication failed using login context 'ZkClient' with exception: {} (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
[2020-09-07 14:11:09,814] ERROR [ZooKeeperClient Kafka server] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2020-09-07 14:11:09,833] INFO EventThread shut down for session: 0x100000039900003 (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,889] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /consumers

---- kafka jaas file ---

KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
debug=true
serviceName="kafka"
keyTab="/home/kafka/kafka.service.keytab"
principal="kafka/broker0@KAFKA.SECURE";
};
// ZooKeeper client authentication
ZkClient{
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
debug=true
serviceName="zookeeper"
keyTab="/home/kafka/kafka.service.keytab"
principal="kafka/broker0@KAFKA.SECURE";
};

-- zookeeper jaas--

QuorumServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/home/zookeeper/zookeeper.service.keytab"
storeKey=true
useTicketCache=false
debug=false
principal="zookeeper/broker0@EXAMPLE.COM";
};

QuorumLearner {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/home/zookeeper/zookeeper.service.keytab"
storeKey=true
useTicketCache=false
debug=false
principal="zookeeper/broker0@EXAMPLE.COM";
};

Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
storeKey=true
debug=true
keytab="/home/zookeeper/zookeeper.service.keytab"
principal="zookeeper/broker0@KAFKA.SECURE";
};

Thanks,

Chiranjeevi

[#2 · Accepted solution]

I am able to fix this issue, posting the resolution here just in case if it helps anyone. I was using the embedded zookeeper which comes with Kafka. I see the below code is missing in the zookeeper server start script which will initialize the environment for the zookeeper. After adding the below code, the jvm process is able to pick the jaas file properly and sasl configuration is complete. After this Kafka is able to SASL auth to zookeeper without any issues.

if [ "x$KAFKA_OPTS" = "x" ]; then

    export KAFKA_OPTS="-Djava.security.auth.login.config=/home/zookeeper/zookeeper_jaas.conf"

fi