Community Articles

Find and share helpful community-sourced technical articles.
Celebrating as our community reaches 100,000 members! Thank you!
Labels (2)

Pragmatic Kafka Security 0.9, Setup and Java Producer

  • Jump Start Kafka Security implementation.
  • SSL for Authentication, ACL for Authorization
  • The below steps will provide 4 vms with Kafka Zookeeper installed on all of them via Vagrant
  • SSL authentication will be enabled between the Consumers and Brokers
  • ACL is also enabled



  • New Console Consumer adds by default.
  • Main Commands. Vagrant suspend, Vagrant resume --no-provision , Vagrant destroy
  • To clean up all, just run vagrant destroy -f (everything will get cleaned)


Installation and Running (install commands)


  • git clone
  • cd kafka-security-demo
  • Run and go for coffee or just read along documentation ( 10 - 15 min)
    • (internally runs sh /vagrant/data/ => update software, install java, kafka, zoo)
    • (internally runs sh /vagrant/data/ => Become CA root, generate public and private key)
    • (internally runs sh /vagrant/data/ => generates ca request and puts in shared folder /vagrant/data)

Step 2 :

  • open a new terminal same path ($PWD), run the below commands
    • vagrant ssh c7001 => Login to Box
    • sudo su => Login as root
    • sh /vagrant/data/ =>Sign the cert-request from C700* and put signed request to /vagrant/data/ and also copy the root-ca
      • Edit the file, update the hostname for each client, default to c7002
  • In the New terminal same path ($PWD),
    • vagrant ssh c7002,3,4 (one bo) =>Login to Box
    • sudo su =>Login as root
    • sh /vagrant/data/ =>Install both root Ca and signed Certificate

Step 3 :

  • start Zookeeper on server
    • sh zookeeper-3.4.8/bin/ start
  • start kafka on server
sh kafka_2.11- --authorizer-properties zookeeper.connect=localhost:2181 --operation All --allow-principal User:*--allow-host --add --cluster
    This will allow local server machine all ACL
nohup sh kafka_2.11- kafka_2.11- & (Run in background)
  • Create Topic
sh kafka_2.11- --create --zookeeper --replication-factor 1 --partitions 1 --topic test
sh kafka_2.11- --authorizer-properties zookeeper.connect=localhost:2181 --operation Write --allow-principal User:* --allow-host --add --topic test
  • Enter data, Two Options
    • manual Producer
        sh kafka_2.11- --broker-list --topic test --producer.config securityDemo/
* Java Producer, Go outside the Vagrant box
           mvn clean package
         cp src/main/resources/Producer.Properties data/
         cp target/kafka-security-demo-1.0.0-jar-with-dependencies.jar data/
    * Login into Server, Vagrant ssh c7001 and run below
         java -cp /vagrant/data/kafka-security-demo-1.0.0-jar-with-dependencies.jar /vagrant/data/Producer.Properties
* Allow c7002 to read data
        sh kafka_2.11- --authorizer-properties zookeeper.connect=localhost:2181 --operation Read --allow-principal User:* --allow-host --add --topic test --group group102
  • Consumer
    • On the client c7002
    • Add Consumer group
      • vim securityDemo/
    • Run the new consumer
     sh kafka_2.11- --bootstrap-server  --topic test --from-beginning --new-consumer --consumer.conf securityDemo/

List important functions with example commands

  • sh kafka_2.11- --authorizer-properties zookeeper.connect=localhost:2181 --list
  • sh kafka_2.11- --list --zookeeper localhost:2181


Narendra Bidari Mahipal

References, Additional Information