How to auto create topics in Ranger enabled Kafka cluster

[#1]

In the current version of Kafka, when Kafka cluster is enabled with ranger authorizer(authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer) it is not possible to auto create topics as a non super user even if the auto create topic flag is set to true. In other words, Kafka create topic authorization can not be done at a topic level.

For example, create a ranger policy as below,

Topic AutoCreateTopic_Test* with all permissions to a non super user. Run the command line Kafka producer script to

create a non existing topic,

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list < > --topic AutoCreateTopic_Test01--security-protocol PLAINTEXTSASL

[2017-02-24 19:10:30,232] WARN Error while fetching metadata [{TopicMetadata for topic test4 -> No partition metadata for topic test4 due to kafka.common.TopicAuthorizationException}] for topic [test4]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo) [2017-02-24 19:10:30,706] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread) kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:91) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68) at scala.collection.immutable.Stream.foreach(Stream.scala:547) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67) at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)

This is because Topic creation is currently a cluster level privilege. Thus it requires access privileges over all topics in a cluster, i.e. *.

Workaround:

Simple workaround is to add a ranger policy with create permissions over all topics in a cluster, i.e. *.

Create a new ranger policy like shown above. PlaceHolderTopicName as the name suggest it is just a random topic name to distinguish this ranger policy with the default ranger policy associated with Topic “*”. Add users, groups and give only create permissions. Once the policy get refreshed, users in this policy should be able to auto create topics.

Roadmap items:

Please find the apache kafka jira’s related to address this limitation in the future kafka releases.

https://issues.apache.org/jira/browse/KAFKA-2945

https://issues.apache.org/jira/browse/KAFKA-2946

References:

https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin#KafkaPlugin-WhydoIhavetograntcreatea...?