Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure is thrown while creating kafka topic

[#1]

when i try to create topic in kafka using non kafka admin user with following command:

usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper <zk_url> --replication-factor 1 --partitions 1 --topic kafka_topic

[2016-11-09 09:20:32,716] WARN Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. (org.apache.zookeeper.client.ZooKeeperSaslClient)[2016-11-09 09:20:32,719] WARN SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failureat org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:923)at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1230)at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:75)at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)at kafka.admin.TopicCommand.main(TopicCommand.scala)

but when i run same command as kafka user then it works, any clue on this ?

Note: this is a secure cluster

[#2 · Accepted solution]

If the cluster is kerberized only one user can create new topic with command:

bin/kafka-topics.sh --create --zookeeper <you zk node> --replication-factor 1 --partitions 1 --topic kafka_topic

This user is the user who runs the kafka broker, this is a zookeeper security limitation but there is a property in kafka auto.create.topics.enable, if this one is set to true the required topic will be created if the user produce/consume message.

bin/kafka-console-producer.sh --broker-list broker address --topic kafka_topic --security-protocol SASL_PLAINTEXT

This command will create the kafka_topic, with different users.