Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Topic creation and deletion are not protected after enabling Kerberos in Kafka

Solved Go to solution

Topic creation and deletion are not protected after enabling Kerberos in Kafka

Explorer

I have enabled Kerberos authentication for Kafka as the documentation suggests and indeed producing to topics and consuming from topics requires authentication. Surprisingly, topic creation and deletion do not require authentication. Could somebody tell me whether this goes wrong ? Really appreciate it.

 

CDH version: 5.15.1

CDK version: 4.1.0

test command:

 

 

kafka-topics --create --zookeeper <zookeeper-host>:2181 --replication-factor 2 --partitions 3 --topic test2
kafka-topics --delete --zookeeper <zookeeper-host>:2181 --topic test2

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Cloudera Employee

Hi @iamabug 

It's a known limitation in Kafka where the kafka-topics tool communicates directly with Zookeeper. When you create a topic, all the tool does is connect to Zookeeper, creates a znode representing this topic and then sets some data as a JSON string (the metadata for the topic).

There has been work to develop Java admin clients which made some progress:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+...

However, all that's left is to have command line tools that leverage those Java APIs and that's a work in progress:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrativ...


6 REPLIES 6

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Cloudera Employee

Hi @iamabug 

It's a known limitation in Kafka where the kafka-topics tool communicates directly with Zookeeper. When you create a topic, all the tool does is connect to Zookeeper, creates a znode representing this topic and then sets some data as a JSON string (the metadata for the topic).

There has been work to develop Java admin clients which made some progress:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+...

However, all that's left is to have command line tools that leverage those Java APIs and that's a work in progress:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrativ...


Highlighted

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Mentor

@iamabug 

There is a lot more than just kerberizing the cluster and you are good to go. Have you enabled SSL also? Can you share a tokenized version of the below files? Basically, the ACL in zk is the key to who can do what and usually the Kafka admin is the only one allowed!

 

  • server.properties [listeners, advertised.listeners,authorizer.class.name,sasl.enabled.mechanism and super.users]
  • Kafka_server_jaas.conf
  • Kafka_client_jaas.conf
  • kafka_client_kerberos.properties

Hope that helps

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Explorer

Thanks for your answer. I now believe that ACL in Zookeeper may be the solution here.

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Mentor

@iamabug 

Are you now comfortable proceeding? If you need some help don't hesitate to ask.

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Explorer

It's really nice of you. I would definitely ask for your help when something tricky comes up. Thank you very much.

Re: Topic creation and deletion are not protected after enabling Kerberos in Kafka

Explorer

Thanks.