In the current
version of Kafka, when Kafka cluster is enabled with ranger
authorizer(authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)
it is not possible to auto create topics as a non super user even if the auto
create topic flag is set to true. In other words, Kafka create topic authorization
can not be done at a topic level. For example,
create a ranger policy as below, Topic
AutoCreateTopic_Test* with all permissions to a non super user. Run the command line
Kafka producer script to create a non existing topic, /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
--broker-list < > --topic AutoCreateTopic_Test01--security-protocol PLAINTEXTSASL [2017-02-24 19:10:30,232] WARN Error
while fetching metadata [{TopicMetadata for topic test4 ->
No partition metadata for topic test4 due to
kafka.common.TopicAuthorizationException}] for topic [test4]: class
kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionInfo)
[2017-02-24 19:10:30,706] ERROR Error in handling batch of 1 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:91)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
at scala.collection.immutable.Stream.foreach(Stream.scala:547)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45) This is
because Topic creation is currently a cluster level privilege. Thus it
requires access privileges over all topics in a cluster, i.e. *. Workaround: Simple
workaround is to add a ranger policy with create permissions over all topics
in a cluster, i.e. *. Create a new
ranger policy like shown above. PlaceHolderTopicName as the name suggest it is
just a random topic name to distinguish this ranger policy with the default
ranger policy associated with Topic “*”. Add users, groups and give only create
permissions. Once the policy get refreshed, users in this policy should be able
to auto create topics. Roadmap items: Please find
the apache kafka jira’s related to address this limitation in the future kafka
releases.
https://issues.apache.org/jira/browse/KAFKA-2945 https://issues.apache.org/jira/browse/KAFKA-2946 References: https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin#KafkaPlugin-WhydoIhavetograntcreateaccesstoalltopics(via*)toallowforauto-creationtoworkforproducersand/orconsumers?
... View more
