Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Kafka consumer problem on kerberos enabled cluster

New Contributor

Hi everybody,

I have some problem in consuming from topic on a Kerberos enabled cluster. On the cluster is HDP-2.3.2 (with Kafka 0.8.2).

I can read the data from the topic but when my consumer tries to commit offsets I get the following exception. I tried to authorize my consumer but even the documentation isn't clear about it (first it says that you need READ and DESCRIBE permission but then gives READ and CREATE, on an other topic that mentions before).

2016-09-07 12:23:50 ERROR ZookeeperConsumerConnector:103 - [], exception during autoCommit:
org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /consumers/anp-testgroup/offsets/anp_test/0
        at org.I0Itec.zkclient.exception.ZkException.create(
        at org.I0Itec.zkclient.ZkClient.retryUntilConnected(
        at org.I0Itec.zkclient.ZkClient.writeDataReturnStat(
        at org.I0Itec.zkclient.ZkClient.writeData(
        at org.I0Itec.zkclient.ZkClient.writeData(
        at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:417)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsetToZooKeeper(ZookeeperConsumerConnector.scala:304)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$5.apply(ZookeeperConsumerConnector.scala:338)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$5.apply(ZookeeperConsumerConnector.scala:337)
        at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:337)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:324)
        at kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:292)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:146)
        at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:108)
        at kafka.utils.CoreUtils$$anon$
        at java.util.concurrent.Executors$
        at java.util.concurrent.FutureTask.runAndReset(
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
        at java.util.concurrent.ScheduledThreadPoolExecutor$
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /consumers/anp-testgroup/offsets/anp_test/0
        at org.apache.zookeeper.KeeperException.create(
        at org.apache.zookeeper.KeeperException.create(
        at org.apache.zookeeper.ZooKeeper.setData(
        at org.I0Itec.zkclient.ZkConnection.writeDataReturnStat(
        at org.I0Itec.zkclient.ZkClient$
        at org.I0Itec.zkclient.ZkClient.retryUntilConnected(
        ... 21 more

Any advice on what can cause the problem.




looks like issue is related to ZooKeeper permissions. You can try by creating new consumer group. we can use bin/ to verify the acl on znodes.

View solution in original post



looks like issue is related to ZooKeeper permissions. You can try by creating new consumer group. we can use bin/ to verify the acl on znodes.

New Contributor

Thanks @mkumar. I verified the acl-s and just the kafka service had "cdrwa" authorization anyone else just "r". I set it "cdrwa" to anyone and the problem disappeared. But it's weird that by using the kafka authorizer( you can't resolve this.


Hi, script is used to create the ACLs for kafka users. It's not used for zookeeper acl.

As per design, Only broker users can modify the zookeeper nodes, Others can only read the zk nodes. This is to improve security around zookeeper.

You can also use new consumer API, which does not depend Zookeeper. It is available in HDP 2.5.

ps: you can upvote, If you are satisfied with my answer