Unabke to start brokers and zookeepers with auth issue

[#1]

Hi ,

I have kafka cluster managed by CDH 6.3 built on 3 servers acting as brokers and zookeepers, i have enabled kerberos on the clustser but after that i am not able to start it wit the following error on all brokers:

4:29:50.269 AM WARN ClientCnxn
SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/var/run/cloudera-scm-agent/process/4531-kafka-KAFKA_BROKER/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
4:29:50.269 AM INFO ClientCnxn
Opening socket connection to server xxxxx.xxxx:2181
4:29:50.269 AM ERROR ZooKeeperClient
[ZooKeeperClient] Auth failed.
4:29:50.270 AM WARN ClientCnxn
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1224)
4:29:50.844 AM INFO ZooKeeperClient
[ZooKeeperClient] Closing.
4:29:51.370 AM WARN ClientCnxn
SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/var/run/cloudera-scm-agent/process/4531-kafka-KAFKA_BROKER/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
4:29:51.371 AM INFO ClientCnxn
Opening socket connection to server jxxxx.xxx:2181
4:29:51.371 AM ERROR ZooKeeperClient
[ZooKeeperClient] Auth failed.
4:29:51.473 AM INFO ZooKeeper
Session: 0x0 closed
4:29:51.473 AM INFO ClientCnxn
EventThread shut down
4:29:51.475 AM INFO ZooKeeperClient
[ZooKeeperClient] Closed.
4:29:51.477 AM ERROR KafkaServer
Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:242)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:238)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:238)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:238)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1824)
at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:361)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385)
at kafka.server.KafkaServer.startup(KafkaServer.scala:205)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:42)
at kafka.Kafka$.main(Kafka.scala:75)
at com.cloudera.kafka.wrap.Kafka$$anonfun$1.apply(Kafka.scala:92)
at com.cloudera.kafka.wrap.Kafka$$anonfun$1.apply(Kafka.scala:92)
at com.cloudera.kafka.wrap.Kafka$.runMain(Kafka.scala:103)
at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:95)
at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
4:29:51.478 AM INFO KafkaServer
shutting down
4:29:51.481 AM INFO KafkaServer
shut down completed
4:29:51.481 AM ERROR KafkaServerStartable
Exiting Kafka.

Noting that in the path /var/run/cloudera-scm/process/broker and zookeeper i can find the keytabs and jaas files and i can make kinit successfully.

What causing this ? and what should i do or re configure???

[#2]

Can you share the exact steps/list of configuration you changed, to configure kerberos in kafka?

During this time of failure in broker - What is the exact error you notice on zookeeper side?

4:29:51.371 AM ERROR ZooKeeperClient [ZooKeeperClient] Auth failed.

Did you tweak any configuration on zookeeper too?

[#3]

Thank you for your reply ,

I followed cloudera's documentation to configure kafkas kerberos ,

1. From Cloudera Manager, navigate to Kafka > Configurations. Set SSL client authentication to none. Set Inter Broker Protocol to SASL_PLAINTEXT.
2. on each broker : Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties:

   listeners=SASL_PLAINTEXT://broker1FQDN:9093
   advertised.listeners=SASL_PLAINTEXT://broker1FQDN:9093

3. on zookeeper config:

   check the option : Enable Kerberos Authentication
   check the option : Enable Server to Server SASL Authentication

the errors on ZK side are the following :

ZooKeeper-2
ZooKeeper Canary Suppress...
Server (broker1FQDN)
Quorum Membership Suppress...
Server (broker1FQDN)
Quorum Membership Suppress...
Server (broker3FQDN)
Quorum Membership

and in the log:
java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Clock skew too great (37)
at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:137)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:118)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
2020-04-27 11:48:57,528 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration from: /var/run/cloudera-scm-agent/process/4648-zookeeper-server/zoo.cfg
2020-04-27 11:48:57,540 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to majority quorums
2020-04-27 11:48:57,544 INFO org.apache.zookeeper.server.DatadirCleanupManager: autopurge.snapRetainCount set to 5
2020-04-27 11:48:57,544 INFO org.apache.zookeeper.server.DatadirCleanupManager: autopurge.purgeInterval set to 24
2020-04-27 11:48:57,544 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task started.
2020-04-27 11:48:57,552 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task completed.
2020-04-27 11:48:57,556 INFO org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum peer
2020-04-27 12:27:59,649 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally

please help.