Community Articles

Find and share helpful community-sourced technical articles.
Celebrating as our community reaches 100,000 members! Thank you!
Labels (2)

In the current version of Kafka, when Kafka cluster is enabled with ranger authorizer( it is not possible to auto create topics as a non super user even if the auto create topic flag is set to true. In other words, Kafka create topic authorization can not be done at a topic level.

For example, create a ranger policy as below,


Topic AutoCreateTopic_Test* with all permissions to a non super user. Run the command line Kafka producer script to

create a non existing topic,

/usr/hdp/current/kafka-broker/bin/ --broker-list < > --topic AutoCreateTopic_Test01--security-protocol PLAINTEXTSASL

[2017-02-24 19:10:30,232] WARN Error while fetching metadata [{TopicMetadata for topic test4 -> No partition metadata for topic test4 due to kafka.common.TopicAuthorizationException}] for topic [test4]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) [2017-02-24 19:10:30,706] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread) kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:91) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68) at scala.collection.immutable.Stream.foreach(Stream.scala:547) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67) at

This is because Topic creation is currently a cluster level privilege. Thus it requires access privileges over all topics in a cluster, i.e. *.


Simple workaround is to add a ranger policy with create permissions over all topics in a cluster, i.e. *.


Create a new ranger policy like shown above. PlaceHolderTopicName as the name suggest it is just a random topic name to distinguish this ranger policy with the default ranger policy associated with Topic “*”. Add users, groups and give only create permissions. Once the policy get refreshed, users in this policy should be able to auto create topics.

Roadmap items:

Please find the apache kafka jira’s related to address this limitation in the future kafka releases.