Pragmatic Kafka Security Setup 0.9, Java Producer Code

[#1]

Pragmatic Kafka Security 0.9, Setup and Java Producer

• Jump Start Kafka Security implementation.
• SSL for Authentication, ACL for Authorization
• The below steps will provide 4 vms with Kafka Zookeeper installed on all of them via Vagrant
• SSL authentication will be enabled between the Consumers and Brokers
• ACL is also enabled

Prerequisites

• Vagrant : Vagrant installation https://www.vagrantup.com/downloads.html
• Maven
• Basic Java / Kafka Understanding.
• git

Note

• New Console Consumer adds Group.id by default.
• Main Commands. Vagrant suspend, Vagrant resume --no-provision , Vagrant destroy
• To clean up all, just run vagrant destroy -f (everything will get cleaned)

Image

Installation and Running (install commands)

Step1

• git clone https://github.com/Symantec/kafka-security-0.9.git
• cd kafka-security-demo
• Run Start.sh and go for coffee or just read along documentation ( 10 - 15 min)
  • (internally runs sh /vagrant/data/step1-all.sh => update software, install java, kafka, zoo)
  • (internally runs sh /vagrant/data/step2server.sh => Become CA root, generate public and private key)
  • (internally runs sh /vagrant/data/step3client.sh => generates ca request and puts in shared folder /vagrant/data)

Step 2 :

• open a new terminal same path ($PWD), run the below commands
  • vagrant ssh c7001 => Login to Box
  • sudo su => Login as root
  • sh /vagrant/data/step4Server.sh =>Sign the cert-request from C700* and put signed request to /vagrant/data/ and also copy the root-ca
    • Edit the file, update the hostname for each client, default to c7002
• In the New terminal same path ($PWD),
  • vagrant ssh c7002,3,4 (one bo) =>Login to Box
  • sudo su =>Login as root
  • sh /vagrant/data/step5client.sh =>Install both root Ca and signed Certificate

Step 3 :

• start Zookeeper on server
  • sh zookeeper-3.4.8/bin/zkServer.sh start
• start kafka on server

sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation All --allow-principal User:*--allow-host 192.168.70.101 --add --cluster
    This will allow local server machine all ACL
nohup sh kafka_2.11-0.9.0.1/bin/kafka-server-start.sh kafka_2.11-0.9.0.1/config/server.properties & (Run in background)

• Create Topic

sh kafka_2.11-0.9.0.1/bin/kafka-topics.sh --create --zookeeper 192.168.70.101:2181 --replication-factor 1 --partitions 1 --topic test
sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation Write --allow-principal User:* --allow-host 192.168.70.101 --add --topic test

• Enter data, Two Options
  • manual Producer

        sh kafka_2.11-0.9.0.1/bin/kafka-console-producer.sh --broker-list 192.168.70.101:9093 --topic test --producer.config securityDemo/producer.properties

* Java Producer, Go outside the Vagrant box

           mvn clean package

         cp src/main/resources/Producer.Properties data/
         cp target/kafka-security-demo-1.0.0-jar-with-dependencies.jar data/

    * Login into Server, Vagrant ssh c7001 and run below

         java -cp /vagrant/data/kafka-security-demo-1.0.0-jar-with-dependencies.jar com.symantec.cpe.KafkaProducer /vagrant/data/Producer.Properties

* Allow c7002 to read data

        sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --operation Read --allow-principal User:* --allow-host 192.168.70.102 --add --topic test --group group102

• Consumer
  • On the client c7002
  • Add Consumer group
    • vim securityDemo/producer.properties
    • group.id=group102
  • Run the new consumer

     sh kafka_2.11-0.9.0.1/bin/kafka-console-consumer.sh --bootstrap-server c7001.symantec.dev.com:9093  --topic test --from-beginning --new-consumer --consumer.conf securityDemo/producer.properties

List important functions with example commands

• sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --list
• sh kafka_2.11-0.9.0.1/bin/kafka-topics.sh --list --zookeeper localhost:2181

Contributions

Narendra Bidari Mahipal

References, Additional Information

• https://msdn.microsoft.com/en-us/library/windows/desktop/aa382479(v=vs.85).aspx
• http://stackoverflow.com/questions/35653128/kafka-ssl-security-setup-causing-issue
• http://kafka.apache.org/documentation.html#security_ssl
• https://www.sslshopper.com/article-most-common-java-keytool-keystore-commands.html
• https://chrisjean.com/change-timezone-in-centos/
• http://docs.confluent.io/2.0.0/kafka/ssl.html
• https://developer.ibm.com/messaging/2016/03/03/message-hub-kafka-java-api/
• http://kafka.apache.org/documentation.html#security_authz
• https://cwiki.apache.org/confluence/display/KAFKA/Security