Support Questions
Find answers, ask questions, and share your expertise

Topic creation and deletion are not protected after enabling Kerberos in Kafka

Solved Go to solution

Topic creation and deletion are not protected after enabling Kerberos in Kafka

Contributor

I have enabled Kerberos authentication for Kafka as the documentation suggests and indeed producing to topics and consuming from topics requires authentication. Surprisingly, topic creation and deletion do not require authentication. Could somebody tell me whether this goes wrong ? Really appreciate it.

 

CDH version: 5.15.1

CDK version: 4.1.0

test command:

 

 

kafka-topics --create --zookeeper <zookeeper-host>:2181 --replication-factor 2 --partitions 3 --topic test2
kafka-topics --delete --zookeeper <zookeeper-host>:2181 --topic test2

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Cloudera Employee

Hi @iamabug 

It's a known limitation in Kafka where the kafka-topics tool communicates directly with Zookeeper. When you create a topic, all the tool does is connect to Zookeeper, creates a znode representing this topic and then sets some data as a JSON string (the metadata for the topic).

There has been work to develop Java admin clients which made some progress:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+...

However, all that's left is to have command line tools that leverage those Java APIs and that's a work in progress:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrativ...


View solution in original post

8 REPLIES 8

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Cloudera Employee

Hi @iamabug 

It's a known limitation in Kafka where the kafka-topics tool communicates directly with Zookeeper. When you create a topic, all the tool does is connect to Zookeeper, creates a znode representing this topic and then sets some data as a JSON string (the metadata for the topic).

There has been work to develop Java admin clients which made some progress:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+...

However, all that's left is to have command line tools that leverage those Java APIs and that's a work in progress:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrativ...


View solution in original post

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Mentor

@iamabug 

There is a lot more than just kerberizing the cluster and you are good to go. Have you enabled SSL also? Can you share a tokenized version of the below files? Basically, the ACL in zk is the key to who can do what and usually the Kafka admin is the only one allowed!

 

  • server.properties [listeners, advertised.listeners,authorizer.class.name,sasl.enabled.mechanism and super.users]
  • Kafka_server_jaas.conf
  • Kafka_client_jaas.conf
  • kafka_client_kerberos.properties

Hope that helps

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Contributor

Thanks for your answer. I now believe that ACL in Zookeeper may be the solution here.

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Mentor

@iamabug 

Are you now comfortable proceeding? If you need some help don't hesitate to ask.

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Contributor

It's really nice of you. I would definitely ask for your help when something tricky comes up. Thank you very much.

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

@Shelton I have the same problem but with cloudera, do you know what procedure I should follow to configure the zookeeper ACL but with kafka and sentry? Thank you

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

@WilsonLozano,

 

As this thread is older and was marked 'Solved back in August of 2019 you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment, version of CDH, etc. that could aid others in providing a more accurate answer to your question. 

 

 

Bill Brooks, Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Contributor

Thanks.