Community Articles

Find and share helpful community-sourced technical articles.
Labels (2)
Cloudera Employee

In the current version of Kafka, when Kafka cluster is enabled with ranger authorizer( it is not possible to auto create topics as a non super user even if the auto create topic flag is set to true. In other words, Kafka create topic authorization can not be done at a topic level.

For example, create a ranger policy as below,


Topic AutoCreateTopic_Test* with all permissions to a non super user. Run the command line Kafka producer script to

create a non existing topic,

/usr/hdp/current/kafka-broker/bin/ --broker-list < > --topic AutoCreateTopic_Test01--security-protocol PLAINTEXTSASL

[2017-02-24 19:10:30,232] WARN Error while fetching metadata [{TopicMetadata for topic test4 -> No partition metadata for topic test4 due to kafka.common.TopicAuthorizationException}] for topic [test4]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) [2017-02-24 19:10:30,706] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread) kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:91) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68) at scala.collection.immutable.Stream.foreach(Stream.scala:547) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67) at

This is because Topic creation is currently a cluster level privilege. Thus it requires access privileges over all topics in a cluster, i.e. *.


Simple workaround is to add a ranger policy with create permissions over all topics in a cluster, i.e. *.


Create a new ranger policy like shown above. PlaceHolderTopicName as the name suggest it is just a random topic name to distinguish this ranger policy with the default ranger policy associated with Topic “*”. Add users, groups and give only create permissions. Once the policy get refreshed, users in this policy should be able to auto create topics.

Roadmap items:

Please find the apache kafka jira’s related to address this limitation in the future kafka releases.


Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎08-17-2019 01:34 PM
Updated by:
Top Kudoed Authors