Community Articles
Find and share helpful community-sourced technical articles
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)
New Contributor

Pragmatic Kafka Security 0.9, Setup and Java Producer

  • Jump Start Kafka Security implementation.
  • SSL for Authentication, ACL for Authorization
  • The below steps will provide 4 vms with Kafka Zookeeper installed on all of them via Vagrant
  • SSL authentication will be enabled between the Consumers and Brokers
  • ACL is also enabled



  • New Console Consumer adds by default.
  • Main Commands. Vagrant suspend, Vagrant resume --no-provision , Vagrant destroy
  • To clean up all, just run vagrant destroy -f (everything will get cleaned)


Installation and Running (install commands)


  • git clone
  • cd kafka-security-demo
  • Run and go for coffee or just read along documentation ( 10 - 15 min)
    • (internally runs sh /vagrant/data/ => update software, install java, kafka, zoo)
    • (internally runs sh /vagrant/data/ => Become CA root, generate public and private key)
    • (internally runs sh /vagrant/data/ => generates ca request and puts in shared folder /vagrant/data)

Step 2 :

  • open a new terminal same path ($PWD), run the below commands
    • vagrant ssh c7001 => Login to Box
    • sudo su => Login as root
    • sh /vagrant/data/ =>Sign the cert-request from C700* and put signed request to /vagrant/data/ and also copy the root-ca
      • Edit the file, update the hostname for each client, default to c7002
  • In the New terminal same path ($PWD),
    • vagrant ssh c7002,3,4 (one bo) =>Login to Box
    • sudo su =>Login as root
    • sh /vagrant/data/ =>Install both root Ca and signed Certificate

Step 3 :

  • start Zookeeper on server
    • sh zookeeper-3.4.8/bin/ start
  • start kafka on server
sh kafka_2.11- --authorizer-properties zookeeper.connect=localhost:2181 --operation All --allow-principal User:*--allow-host --add --cluster
    This will allow local server machine all ACL
nohup sh kafka_2.11- kafka_2.11- & (Run in background)
  • Create Topic
sh kafka_2.11- --create --zookeeper --replication-factor 1 --partitions 1 --topic test
sh kafka_2.11- --authorizer-properties zookeeper.connect=localhost:2181 --operation Write --allow-principal User:* --allow-host --add --topic test
  • Enter data, Two Options
    • manual Producer
        sh kafka_2.11- --broker-list --topic test --producer.config securityDemo/
* Java Producer, Go outside the Vagrant box
           mvn clean package
         cp src/main/resources/Producer.Properties data/
         cp target/kafka-security-demo-1.0.0-jar-with-dependencies.jar data/
    * Login into Server, Vagrant ssh c7001 and run below
         java -cp /vagrant/data/kafka-security-demo-1.0.0-jar-with-dependencies.jar /vagrant/data/Producer.Properties
* Allow c7002 to read data
        sh kafka_2.11- --authorizer-properties zookeeper.connect=localhost:2181 --operation Read --allow-principal User:* --allow-host --add --topic test --group group102
  • Consumer
    • On the client c7002
    • Add Consumer group
      • vim securityDemo/
    • Run the new consumer
     sh kafka_2.11- --bootstrap-server  --topic test --from-beginning --new-consumer --consumer.conf securityDemo/

List important functions with example commands

  • sh kafka_2.11- --authorizer-properties zookeeper.connect=localhost:2181 --list
  • sh kafka_2.11- --list --zookeeper localhost:2181


Narendra Bidari Mahipal

References, Additional Information

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎03-19-2016 02:07 AM
Updated by:
Top Kudoed Authors