Kafka consumer problem on kerberos enabled cluster

[#1]

Hi everybody,

I have some problem in consuming from topic on a Kerberos enabled cluster. On the cluster is HDP-2.3.2 (with Kafka 0.8.2).

I can read the data from the topic but when my consumer tries to commit offsets I get the following exception. I tried to authorize my consumer but even the documentation isn't clear about it (first it says that you need READ and DESCRIBE permission but then gives READ and CREATE, on an other topic that mentions before).

2016-09-07 12:23:50 ERROR ZookeeperConsumerConnector:103 - [anp-testgroup_mybroker.host.com-1473243827746-c4749523], exception during autoCommit:
org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /consumers/anp-testgroup/offsets/anp_test/0
        at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:68)
        at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:981)
        at org.I0Itec.zkclient.ZkClient.writeDataReturnStat(ZkClient.java:1121)
        at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:1116)
        at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:1085)
        at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:417)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsetToZooKeeper(ZookeeperConsumerConnector.scala:304)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$5.apply(ZookeeperConsumerConnector.scala:338)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$5.apply(ZookeeperConsumerConnector.scala:337)
        at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:337)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:324)
        at kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:292)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:146)
        at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:108)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /consumers/anp-testgroup/offsets/anp_test/0
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
        at org.I0Itec.zkclient.ZkConnection.writeDataReturnStat(ZkConnection.java:126)
        at org.I0Itec.zkclient.ZkClient$14.call(ZkClient.java:1125)
        at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:971)
        ... 21 more

Any advice on what can cause the problem.

Thanks!

[#2 · Accepted solution]

looks like issue is related to ZooKeeper permissions. You can try by creating new consumer group. we can use bin/zookeeper-shell.sh to verify the acl on znodes.

[#3]

Thanks @mkumar. I verified the acl-s and just the kafka service had "cdrwa" authorization anyone else just "r". I set it "cdrwa" to anyone and the problem disappeared. But it's weird that by using the kafka authorizer(kafka-acls.sh) you can't resolve this.

[#4]

Hi,

kafka-acls.sh script is used to create the ACLs for kafka users. It's not used for zookeeper acl.

As per design, Only broker users can modify the zookeeper nodes, Others can only read the zk nodes. This is to improve security around zookeeper.

You can also use new consumer API, which does not depend Zookeeper. It is available in HDP 2.5.

ps: you can upvote, If you are satisfied with my answer