Skip to main content
Narendra_Bidari
Narendra_Bidari
Silver ⭐️⭐️
March 19, 2016

Pragmatic Kafka Security Setup 0.9, Java Producer Code

  • October 5, 2026 at 9:41 AM
  • 0 replies
  • 3546 views
  • Translate All

Pragmatic Kafka Security 0.9, Setup and Java Producer

  • Jump Start Kafka Security implementation.
  • SSL for Authentication, ACL for Authorization
  • The below steps will provide 4 vms with Kafka Zookeeper installed on all of them via Vagrant
  • SSL authentication will be enabled between the Consumers and Brokers
  • ACL is also enabled

Prerequisites

Note

  • New Console Consumer adds Group.id by default.
  • Main Commands. Vagrant suspend, Vagrant resume --no-provision , Vagrant destroy
  • To clean up all, just run vagrant destroy -f (everything will get cleaned)

Image

Installation and Running (install commands)

Step1

  • git clone https://github.com/Symantec/kafka-security-0.9.git
  • cd kafka-security-demo
  • Run Start.sh and go for coffee or just read along documentation ( 10 - 15 min)
    • (internally runs sh /vagrant/data/step1-all.sh => update software, install java, kafka, zoo)
    • (internally runs sh /vagrant/data/step2server.sh => Become CA root, generate public and private key)
    • (internally runs sh /vagrant/data/step3client.sh => generates ca request and puts in shared folder /vagrant/data)

Step 2 :

  • open a new terminal same path ($PWD), run the below commands
    • vagrant ssh c7001 => Login to Box
    • sudo su => Login as root
    • sh /vagrant/data/step4Server.sh =>Sign the cert-request from C700* and put signed request to /vagrant/data/ and also copy the root-ca
      • Edit the file, update the hostname for each client, default to c7002
  • In the New terminal same path ($PWD),
    • vagrant ssh c7002,3,4 (one bo) =>Login to Box
    • sudo su =>Login as root
    • sh /vagrant/data/step5client.sh =>Install both root Ca and signed Certificate

Step 3 :

  • start Zookeeper on server
    • sh zookeeper-3.4.8/bin/zkServer.sh start
  • start kafka on server

sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation All --allow-principal User:*--allow-host 192.168.70.101 --add --cluster
This will allow local server machine all ACL
nohup sh kafka_2.11-0.9.0.1/bin/kafka-server-start.sh kafka_2.11-0.9.0.1/config/server.properties & (Run in background)

  • Create Topic

sh kafka_2.11-0.9.0.1/bin/kafka-topics.sh --create --zookeeper 192.168.70.101:2181 --replication-factor 1 --partitions 1 --topic test
sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation Write --allow-principal User:* --allow-host 192.168.70.101 --add --topic test

  • Enter data, Two Options
    • manual Producer

sh kafka_2.11-0.9.0.1/bin/kafka-console-producer.sh --broker-list 192.168.70.101:9093 --topic test --producer.config securityDemo/producer.properties

* Java Producer, Go outside the Vagrant box

mvn clean package

cp src/main/resources/Producer.Properties data/
cp target/kafka-security-demo-1.0.0-jar-with-dependencies.jar data/

    * Login into Server, Vagrant ssh c7001 and run below

java -cp /vagrant/data/kafka-security-demo-1.0.0-jar-with-dependencies.jar com.symantec.cpe.KafkaProducer /vagrant/data/Producer.Properties

* Allow c7002 to read data

sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation Read --allow-principal User:* --allow-host 192.168.70.102 --add --topic test --group group102

  • Consumer
    • On the client c7002
    • Add Consumer group
      • vim securityDemo/producer.properties
      • group.id=group102
    • Run the new consumer

sh kafka_2.11-0.9.0.1/bin/kafka-console-consumer.sh --bootstrap-server c7001.symantec.dev.com:9093 --topic test --from-beginning --new-consumer --consumer.conf securityDemo/producer.properties

List important functions with example commands

  • sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --list
  • sh kafka_2.11-0.9.0.1/bin/kafka-topics.sh --list --zookeeper localhost:2181

Contributions

Narendra Bidari Mahipal

References, Additional Information

    Terms & ConditionsAccessibility statement