Kafka consumer problem on kerberos enabled cluster

mattyy_007 — Wed, 07 Sep 2016 19:02:21 GMT

Hi everybody,

I have some problem in consuming from topic on a Kerberos enabled cluster. On the cluster is HDP-2.3.2 (with Kafka 0.8.2).

I can read the data from the topic but when my consumer tries to commit offsets I get the following exception. I tried to authorize my consumer but even the documentation isn't clear about it (first it says that you need READ and DESCRIBE permission but then gives READ and CREATE, on an other topic that mentions before).

2016-09-07 12:23:50 ERROR ZookeeperConsumerConnector:103 - [anp-testgroup_mybroker.host.com-1473243827746-c4749523], exception during autoCommit:
org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /consumers/anp-testgroup/offsets/anp_test/0
        at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:68)
        at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:981)
        at org.I0Itec.zkclient.ZkClient.writeDataReturnStat(ZkClient.java:1121)
        at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:1116)
        at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:1085)
        at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:417)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsetToZooKeeper(ZookeeperConsumerConnector.scala:304)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$5.apply(ZookeeperConsumerConnector.scala:338)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$5.apply(ZookeeperConsumerConnector.scala:337)
        at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:337)
        at kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:324)
        at kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:292)
        at kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:146)
        at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:108)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /consumers/anp-testgroup/offsets/anp_test/0
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
        at org.I0Itec.zkclient.ZkConnection.writeDataReturnStat(ZkConnection.java:126)
        at org.I0Itec.zkclient.ZkClient$14.call(ZkClient.java:1125)
        at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:971)
        ... 21 more

Any advice on what can cause the problem.

Thanks!

Re: Kafka consumer problem on kerberos enabled cluster

mkumar2 — Wed, 07 Sep 2016 21:35:25 GMT

looks like issue is related to ZooKeeper permissions. You can try by creating new consumer group. we can use bin/zookeeper-shell.sh to verify the acl on znodes.

Re: Kafka consumer problem on kerberos enabled cluster

mattyy_007 — Thu, 08 Sep 2016 15:28:00 GMT

Thanks @mkumar. I verified the acl-s and just the kafka service had "cdrwa" authorization anyone else just "r". I set it "cdrwa" to anyone and the problem disappeared. But it's weird that by using the kafka authorizer(kafka-acls.sh) you can't resolve this.

Re: Kafka consumer problem on kerberos enabled cluster

mkumar2 — Thu, 08 Sep 2016 16:06:58 GMT

Hi,

kafka-acls.sh script is used to create the ACLs for kafka users. It's not used for zookeeper acl.

As per design, Only broker users can modify the zookeeper nodes, Others can only read the zk nodes. This is to improve security around zookeeper.

You can also use new consumer API, which does not depend Zookeeper. It is available in HDP 2.5.

ps: you can upvote, If you are satisfied with my answer

question Re: Kafka consumer problem on kerberos enabled cluster in Archives of Support Questions (Read Only)

Kafka consumer problem on kerberos enabled cluster

Re: Kafka consumer problem on kerberos enabled cluster

Re: Kafka consumer problem on kerberos enabled cluster

Re: Kafka consumer problem on kerberos enabled cluster