Community Articles
Find and share helpful community-sourced technical articles
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)
New Contributor

In the current version of Kafka, when Kafka cluster is enabled with ranger authorizer( it is not possible to auto create topics as a non super user even if the auto create topic flag is set to true. In other words, Kafka create topic authorization can not be done at a topic level.

For example, create a ranger policy as below,


Topic AutoCreateTopic_Test* with all permissions to a non super user. Run the command line Kafka producer script to

create a non existing topic,

/usr/hdp/current/kafka-broker/bin/ --broker-list < > --topic AutoCreateTopic_Test01--security-protocol PLAINTEXTSASL

[2017-02-24 19:10:30,232] WARN Error while fetching metadata [{TopicMetadata for topic test4 -> No partition metadata for topic test4 due to kafka.common.TopicAuthorizationException}] for topic [test4]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) [2017-02-24 19:10:30,706] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread) kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:91) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68) at scala.collection.immutable.Stream.foreach(Stream.scala:547) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67) at

This is because Topic creation is currently a cluster level privilege. Thus it requires access privileges over all topics in a cluster, i.e. *.


Simple workaround is to add a ranger policy with create permissions over all topics in a cluster, i.e. *.


Create a new ranger policy like shown above. PlaceHolderTopicName as the name suggest it is just a random topic name to distinguish this ranger policy with the default ranger policy associated with Topic “*”. Add users, groups and give only create permissions. Once the policy get refreshed, users in this policy should be able to auto create topics.

Roadmap items:

Please find the apache kafka jira’s related to address this limitation in the future kafka releases.


Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 01:34 PM
Updated by:
Top Kudoed Authors