Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka broker SASL connection failing with Zookeeper

Solved Go to solution

Kafka broker SASL connection failing with Zookeeper

New Contributor

Dear experts,

 

I have installed apache kafka 2.4 on one node. I am having SSL and SASL(Kerberos) enabled for kafka broker and now enabled SASL for zookeeper. However when starting the broker , i am getting the below error, could you please help on this ?

 

--error log---

 

[2020-09-07 14:11:09,761] DEBUG Client principal is "kafka/broker0@KAFKA.SECURE". (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,761] DEBUG Server principal is "krbtgt/KAFKA.SECURE@KAFKA.SECURE". (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,764] INFO TGT valid starting at: Mon Sep 07 14:11:09 IST 2020 (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,765] INFO TGT expires: Tue Sep 08 14:11:09 IST 2020 (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,765] INFO TGT refresh sleeping until: Tue Sep 08 09:30:58 IST 2020 (org.apache.zookeeper.Login)
[2020-09-07 14:11:09,765] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-09-07 14:11:09,766] DEBUG creating sasl client: Client=kafka/broker0@KAFKA.SECURE;service=kafka;serviceHostname=broker0 (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-09-07 14:11:09,773] INFO Opening socket connection to server broker0/X.X.X.X:2181. Will attempt to SASL-authenticate using Login Context section 'ZkClient' (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,778] INFO Socket connection established, initiating session, client: /X.X.X.X:54728, server: broker0/X.X.X.X:2181 (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,780] DEBUG Session establishment request sent on broker0/X.X.X.X:2181 (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,785] INFO Session establishment complete on server broker0/X.X.X.X:2181, sessionid = 0x100000039900003, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,786] DEBUG ClientCnxn:sendSaslPacket:length=0 (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-09-07 14:11:09,787] DEBUG saslClient.evaluateChallenge(len=0) (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2020-09-07 14:11:09,789] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2020-09-07 14:11:09,811] ERROR SASL authentication failed using login context 'ZkClient' with exception: {} (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
[2020-09-07 14:11:09,814] ERROR [ZooKeeperClient Kafka server] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2020-09-07 14:11:09,833] INFO EventThread shut down for session: 0x100000039900003 (org.apache.zookeeper.ClientCnxn)
[2020-09-07 14:11:09,889] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /consumers

 

---- kafka jaas file ---

KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
debug=true
serviceName="kafka"
keyTab="/home/kafka/kafka.service.keytab"
principal="kafka/broker0@KAFKA.SECURE";
};
// ZooKeeper client authentication
ZkClient{
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
debug=true
serviceName="zookeeper"
keyTab="/home/kafka/kafka.service.keytab"
principal="kafka/broker0@KAFKA.SECURE";
};

 

-- zookeeper jaas--

 

QuorumServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/home/zookeeper/zookeeper.service.keytab"
storeKey=true
useTicketCache=false
debug=false
principal="zookeeper/broker0@EXAMPLE.COM";
};

QuorumLearner {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/home/zookeeper/zookeeper.service.keytab"
storeKey=true
useTicketCache=false
debug=false
principal="zookeeper/broker0@EXAMPLE.COM";
};

Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
storeKey=true
debug=true
keytab="/home/zookeeper/zookeeper.service.keytab"
principal="zookeeper/broker0@KAFKA.SECURE";
};

 

Thanks,

Chiranjeevi

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Kafka broker SASL connection failing with Zookeeper

New Contributor

I am able to fix this issue, posting the resolution here just in case if it helps anyone. I was using the embedded zookeeper which comes with Kafka. I see the below code is missing in the zookeeper server start script which will initialize the environment for the zookeeper. After adding the below code, the jvm process is able to pick the jaas file properly and sasl configuration is complete. After this Kafka is able to SASL auth to zookeeper without any issues.

 

if [ "x$KAFKA_OPTS" = "x" ]; then

    export KAFKA_OPTS="-Djava.security.auth.login.config=/home/zookeeper/zookeeper_jaas.conf"

fi

View solution in original post

1 REPLY 1
Highlighted

Re: Kafka broker SASL connection failing with Zookeeper

New Contributor

I am able to fix this issue, posting the resolution here just in case if it helps anyone. I was using the embedded zookeeper which comes with Kafka. I see the below code is missing in the zookeeper server start script which will initialize the environment for the zookeeper. After adding the below code, the jvm process is able to pick the jaas file properly and sasl configuration is complete. After this Kafka is able to SASL auth to zookeeper without any issues.

 

if [ "x$KAFKA_OPTS" = "x" ]; then

    export KAFKA_OPTS="-Djava.security.auth.login.config=/home/zookeeper/zookeeper_jaas.conf"

fi

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here