Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Unabke to start brokers and zookeepers with auth issue

avatar
Explorer

Hi ,

 

I have kafka cluster managed by CDH 6.3 built on 3 servers acting as brokers and zookeepers, i have enabled kerberos on the clustser but after that i am not able to start it wit the following error on all brokers:

 

4:29:50.269 AM WARN ClientCnxn
SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/var/run/cloudera-scm-agent/process/4531-kafka-KAFKA_BROKER/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
4:29:50.269 AM INFO ClientCnxn
Opening socket connection to server xxxxx.xxxx:2181
4:29:50.269 AM ERROR ZooKeeperClient
[ZooKeeperClient] Auth failed.
4:29:50.270 AM WARN ClientCnxn
Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1224)
4:29:50.844 AM INFO ZooKeeperClient
[ZooKeeperClient] Closing.
4:29:51.370 AM WARN ClientCnxn
SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/var/run/cloudera-scm-agent/process/4531-kafka-KAFKA_BROKER/jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
4:29:51.371 AM INFO ClientCnxn
Opening socket connection to server jxxxx.xxx:2181
4:29:51.371 AM ERROR ZooKeeperClient
[ZooKeeperClient] Auth failed.
4:29:51.473 AM INFO ZooKeeper
Session: 0x0 closed
4:29:51.473 AM INFO ClientCnxn
EventThread shut down
4:29:51.475 AM INFO ZooKeeperClient
[ZooKeeperClient] Closed.
4:29:51.477 AM ERROR KafkaServer
Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:242)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:238)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:238)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:238)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1824)
at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:361)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385)
at kafka.server.KafkaServer.startup(KafkaServer.scala:205)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:42)
at kafka.Kafka$.main(Kafka.scala:75)
at com.cloudera.kafka.wrap.Kafka$$anonfun$1.apply(Kafka.scala:92)
at com.cloudera.kafka.wrap.Kafka$$anonfun$1.apply(Kafka.scala:92)
at com.cloudera.kafka.wrap.Kafka$.runMain(Kafka.scala:103)
at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:95)
at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
4:29:51.478 AM INFO KafkaServer
shutting down
4:29:51.481 AM INFO KafkaServer
shut down completed
4:29:51.481 AM ERROR KafkaServerStartable
Exiting Kafka.

 

Noting that in the path /var/run/cloudera-scm/process/broker and zookeeper i can find the keytabs and jaas files and i can make kinit successfully.

What causing this ? and what should i do or re configure???

 

2 REPLIES 2

avatar
Master Collaborator

Can you share the exact steps/list of configuration you changed, to configure kerberos in kafka?

 

During this time of failure in broker - What is the exact error you notice on zookeeper side?

4:29:51.371 AM ERROR ZooKeeperClient [ZooKeeperClient] Auth failed.

 

Did you tweak any configuration on zookeeper too?

avatar
Explorer

Thank you for your reply ,

I followed cloudera's documentation to configure kafkas kerberos ,

  1. From Cloudera Manager, navigate to Kafka > Configurations. Set SSL client authentication to none. Set Inter Broker Protocol to SASL_PLAINTEXT.
  2. on each broker : Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties:

    listeners=SASL_PLAINTEXT://broker1FQDN:9093
    advertised.listeners=SASL_PLAINTEXT://broker1FQDN:9093

  3.  

    on zookeeper config:

    check the option : Enable Kerberos Authentication
    check the option : Enable Server to Server SASL Authentication

the errors on ZK side are the following :

ZooKeeper-2
ZooKeeper Canary Suppress...
Server (broker1FQDN)
Quorum Membership Suppress...
Server (broker1FQDN)
Quorum Membership Suppress...
Server (broker3FQDN)
Quorum Membership

and in the log:
java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Clock skew too great (37)
at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:211)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:137)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:118)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
2020-04-27 11:48:57,528 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration from: /var/run/cloudera-scm-agent/process/4648-zookeeper-server/zoo.cfg
2020-04-27 11:48:57,540 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to majority quorums
2020-04-27 11:48:57,544 INFO org.apache.zookeeper.server.DatadirCleanupManager: autopurge.snapRetainCount set to 5
2020-04-27 11:48:57,544 INFO org.apache.zookeeper.server.DatadirCleanupManager: autopurge.purgeInterval set to 24
2020-04-27 11:48:57,544 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task started.
2020-04-27 11:48:57,552 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task completed.
2020-04-27 11:48:57,556 INFO org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum peer
2020-04-27 12:27:59,649 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally

 

please help.