Support Questions

Find answers, ask questions, and share your expertise

Kafka with SSL

avatar
New Contributor

I am trying to configure a brand new Kafka cluster/sandbox with SSL, but I keep getting errors. I apologize for the length of this email, but I've never worked with keystores/certificates before, so while I'm trying to follow the directions here (http://kafka.apache.org/documentation.html#security_ssl), there are a few things I'm doing my best on but don't quite understand. So I'm trying to include below not just exactly what commands I'm running on exactly which nodes, but my interpretation of exactly what they should be doing. I'm also not trying to get client authentication working for the brokers yet - that will be the next step. 🙂

Setup: I have a brand new 3 node Kafka cluster, and I have 1 Edge node I am going to use as a "CA-node".
Step 1: I need to generate an SSL key and certificate for each Kafka broker.
On all 3 brokers:
keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA -storepass test1234 
When it asks for first/last name, I am giving the FQDN of that broker.
I now have a keystore named server.keystore.jks on each broker. Later we will export the certificate from this keystore so it can be signed by the CA
Step 2:
I need to create a CA, which is a public-private key pair plus a certificate. We will use this CA to sign all 3 broker certificates, and as long as all 3 brokers trust the CA, they will be able to trust each other when they connect.
On the Edge Node that will be the CA:
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365 
For common name, I am using the FQDN of server
At this point I have a "ca-key" and "ca-cert" on the Edge node.
Create a "server.truststore.jks" and "client.truststore.jks" by executing the below:
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert 
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
At this point I have a "server.truststore.jks" and a "client.truststore.jks" on the Edge Node, as well "ca-key" and "ca-cert"
Step 3:
Step 1 created a keystore on each machine. Step 2 created a CA on 1 machine (Edge Node). Now I need to sign each certificate from Step 1 with the CA from Step 2.
On all 3 brokers:
Export the certificate from the keystore
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file 
At this point each broker has a local "cert-file" (an exported certificate). I need to sign those with the CA, using the ca-key and ca-cert. But at this point, the ca-key and ca-cert are on the Edge Node/CA, while the 3 individual certificates are on the 3 separate brokers. So, I am going to (one node at a time) copy the certificates from the brokers onto the CA node and run the command there (which seems better than copying a private key around the cluster).
On the Edge/CA node (1 node at a time):
scp <username>@<FQDN of broker node>:/tmp/cert-file . 
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:test1234 
At this point i now have a ca-cert.srl file and a cert-signed file on the Edge/CA node. But the ca-cert (certificate for the CA) + cert-signed (that I just generated) need to be on the broker nodes so I can import them into the individual broker's keystores, so I am going to copy them back there:
On all 3 brokers (1 node at a time):
scp <username>@<FQDN of Edge/CA node>:/tmp/ca-cert . 
scp <username>@<FQDN of Edge/CA node>:/tmp/cert-signed . 
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert  
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed 
From checking the directions, it appears that each broker should have a local file named "server.truststore.jks". But, I don't. Perhaps I followed something too literally? I did create a server.truststore.jks file in Step 2, but it's sitting on the Edge/CA node. This doesn't seem right, so I am going to instead execute the below command (from Step 2) on all 3 brokers. The ca-cert file was already copied to the brokers so this should work.
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert 
I then add the following lines to my server.properties file on each broker:
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 
ssl.endpoint.identification.algorithm=HTTPS 
ssl.key.password=test1234 
ssl.keystore.location=/var/private/ssl/server.keystore.jks 
ssl.keystore.password=test1234 
ssl.truststore.location=/var/private/ssl/server.truststore.jks 
ssl.truststore.password=test1234 
listeners=SSL://<FQDN>:9093 
security.inter.broker.protocol=SSL 
When I start up Kafka after all of the changes above, I don't see any errors in the logs...but I don't see very many logging entries at all. The Kafka process is definitely up on all 3 brokers, but there's just not much in the logs. When I run the suggested:
openssl s_client -debug -connect FQDN:9093 -tls1 
I get something of the form:
<hexdump> 
-----BEGIN CERTIFICATE----- 
... 
-----END CERTIFICATE----- 
...  
Verify return code: 19 (self signed certificate in certificate chain) 
I'm not sure if that "self signed certificate" error means I did something wrong above?
When I create the following properties file for my consumer/producer on my Edge/CA node:
security.protocol=SSL 
ssl.truststore.location=/var/private/ssl/client.truststore.jks 
ssl.truststore.password=test1234 
and start up a producer/consumer quick test:
kafka-console-consumer.sh --bootstrap-server broker1:9093,broker2:9093,broker3:9093 --topic withssl --consumer.config ssl.properties 
kafka-console-producer.sh --broker-list broker1:9093,broker2:9093,broker3:9093  --topic withssl --producer.config ssl.properties 
Both give me the same error:
[2017-08-30 18:07:58,233] WARN Bootstrap broker broker0:9093 disconnected (org.apache.kafka.clients.NetworkClient) 
[2017-08-30 18:07:58,544] WARN Bootstrap broker broker1:9093 disconnected (org.apache.kafka.clients.NetworkClient) 
[2017-08-30 18:07:58,760] WARN Bootstrap broker broker2:9093 disconnected (org.apache.kafka.clients.NetworkClient) 
I'm not sure if my 3 brokers are set up incorrectly, or it my clients just can't connect. Does anyone have any advice?
2 REPLIES 2

avatar
Super Collaborator

It should be like this, in case only the brokers need to authenticate (one-way server side SSL auth) to the clients:

You need:

1. Client side: 1 keystore (client.truststore.jks) containing just the ca_cert you created and was used to sign the broker / server certificates

2. Server side: On each broker 1 keystore (server.keystore.jks) containing both the server specific certificate generated before, ( which should now be signed by the CA) and the certificate of the CA itself.

What is misleading in the Kafka documentation is the first step, where a keystore 'server.keystore.jks' is created in the very first step, only to export the unsigned cert from to actually sign it. This is not the same 'server.keystore.jks' as in 2. above as the it should not contain the unsigned broker cert anymore ! You don't necessarily need a keystore to create a certificate to sign, you can also just create a cert + key, have it signed and then import it into a new keystore.

Also, it makes more sense to me to copy the CA cert onto the brokers then moving it back-and-forth from server -> CA node (to sign) -> back to server (to import into final 'server.keystore.jks')

Hope it makes sense

avatar
Super Collaborator

On rereading the Kafka docs, it a about one and the same 'server.keystore.jks' on the brokers after all. The replacement of the initially generated unsigned broker cert by the signed version of that same cert is done by importing the signed cert with the same alias, which effectively overwrites the unsigned cert by the signed one.