Kafka:- No partition metadata for topic due to kafka.common.TopicAuthorizationException for topic

[#1]

Hi all,

Trying to produce messages but getting the below errors.

[2016-10-10 20:22:10,947] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test11 (kafka.producer.async.DefaultEventHandler) [2016-10-10 20:22:11,049] WARN Error while fetching metadata [{TopicMetadata for topic test11 -> No partition metadata for topic test11 due to kafka.common.TopicAuthorizationException}] for topic [test11]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) [2016-10-10 20:22:11,051] WARN Error while fetching metadata [{TopicMetadata for topic test11 -> No partition metadata for topic test11 due to kafka.common.TopicAuthorizationException}] for topic [test11]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) [2016-10-10 20:22:11,051] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test11 (kafka.producer.async.DefaultEventHandler) [2016-10-10 20:22:11,153] WARN Error while fetching metadata [{TopicMetadata for topic test11 ->

No partition metadata for topic test11 due to kafka.common.TopicAuthorizationException}] for topic [test11]: class kafka.common.TopicAuthorizationException  (kafka.producer.BrokerPartitionInfo)

[2016-10-10 20:22:11,154] ERROR Failed to send requests for topics test11 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler) [2016-10-10 20:22:11,155] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread) kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:91) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68) at scala.collection.immutable.Stream.foreach(Stream.scala:547) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67) at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)

I have disabled the kerberos.

But I guess Kafka was still trying to get authorise from Kerbors.

Please suggest me.

Thank you.

Mohan.V

[#2]

@Mohan V

Can you please provide more details on

• how you created the topic(the exact command)?
• Do you have permissions to access that topic?
  • Is ranger enabled on the cluster? - if yes, please check if you have policy for enabling permissions to create kafka topic.
  • Is the cluster kerberized? If yes - Are you performing the producer operations as a proper user with proper 'kinit'?

There is a similar question asked in the community, please check this also if it helps(https://community.hortonworks.com/questions/42793/kafka-producer-error-no-partition-metadata-for-top.html)..

[#3]

Thanks for the reply Ayub Pathan.

1:- command to create topic

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

and it is created.

2:-

When kerberos was enabled i dont have the permission to create topic in kafka.

But then I disabled kerberos from cluster without any issues. And I am able to create the topic.

3:- No.Ranger is not enabled in my cluster.

And i have already gone through the link that you have mentioned, but I didnt get any solution as there was no proper explanation that how he solved that issue.

Please suggest me.

Mohan.V

[#4]

@Mohan V Not sure what went wrong while creating the topic. It could be related to disabling kerberos, as this might have some stale bits.

Can you check if the topic is actually created using kafka command line tools?

Also, try creating the topic using the FQDN(zookeeper quorum) for zookeeper instead of localhost. for example:

<strong>./kafka-topics.sh --create --zookeeper <host1:port>,<host2:port> --replication-factor 1 --partitions 1 --topic test</strong>

If none of the above helps, can you try deploying a new cluster without kerberos if possible?

[#5]

@Mohan V, @Ayub Khan - i'm facing the same issue, i'm on HDP 2.4

Is the issue resolved ?

[#6]

After disabling kerberos, you may need to check the Zookeeper znode permissions for kafka - possibly some are still set with sasl security which might cause kafka not to work.

[#7]

@cmcbugg,

pls see details of the issue in link ->

https://community.hortonworks.com/questions/68497/kafka-error-while-fetching-metadata-topicmetadata....

Essentially, this is a fresh HDP 2.4 instance, and i've just enabled Ranger-Kafka plugin.

Zookeper nodes permission :

[kafka1@sandbox ~]$ ls -lrt /hadoop/zookeeper/ total 8 -rw-r--r-- 1 root root 1 2016-03-14 14:17 myid drwxr-xr-x 2 zookeeper hadoop 4096 2016-11-26 19:44 version-2

kafka1 User permission (on hdfs) :

[kafka1@sandbox ~]$ hadoop fs -ls /user/ Found 11 items drwxrwx--- - ambari-qa hdfs 0 2016-03-14 14:18 /user/ambari-qa drwxr-xr-x - hcat hdfs 0 2016-03-14 14:23 /user/hcat drwxr-xr-x - hive hdfs 0 2016-03-14 14:23 /user/hive drwxr-xr-x - kafka1 hdfs 0 2016-11-26 20:31 /user/kafka1 drwxr-xr-x - kafka2 hdfs 0 2016-11-26 20:32 /user/kafka2

Any ideas on what needs to be changed, to enable this ?