I am trying to configure a brand new Kafka cluster/sandbox with SSL, but I keep getting errors. I apologize for the length of this email, but I've never worked with keystores/certificates before, so while I'm trying to follow the directions here (http://kafka.apache.org/documentation.html#security_ssl), there are a few things I'm doing my best on but don't quite understand. So I'm trying to include below not just exactly what commands I'm running on exactly which nodes, but my interpretation of exactly what they should be doing. I'm also not trying to get client authentication working for the brokers yet - that will be the next step. 🙂
Setup: I have a brand new 3 node Kafka cluster, and I have 1 Edge node I am going to use as a "CA-node".
Step 1:
I need to generate an SSL key and certificate for each Kafka broker.
On all 3 brokers:
keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA -storepass test1234
When it asks for first/last name, I am giving the FQDN of that broker.
I now have a keystore named server.keystore.jks on each broker. Later we will export the certificate from this keystore so it can be signed by the CA
Step 2:
I need to create a CA, which is a public-private key pair plus a certificate. We will use this CA to sign all 3 broker certificates, and as long as all 3 brokers trust the CA, they will be able to trust each other when they connect.
On the Edge Node that will be the CA:
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
For common name, I am using the FQDN of server
At this point I have a "ca-key" and "ca-cert" on the Edge node.
Create a "server.truststore.jks" and "client.truststore.jks" by executing the below:
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
At this point I have a "server.truststore.jks" and a "client.truststore.jks" on the Edge Node, as well "ca-key" and "ca-cert"
Step 3:
Step 1 created a keystore on each machine. Step 2 created a CA on 1 machine (Edge Node). Now I need to sign each certificate from Step 1 with the CA from Step 2.
On all 3 brokers:
Export the certificate from the keystore
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file
At this point each broker has a local "cert-file" (an exported certificate). I need to sign those with the CA, using the ca-key and ca-cert. But at this point, the ca-key and ca-cert are on the Edge Node/CA, while the 3 individual certificates are on the 3 separate brokers. So, I am going to (one node at a time) copy the certificates from the brokers onto the CA node and run the command there (which seems better than copying a private key around the cluster).
On the Edge/CA node (1 node at a time):
scp <username>@<FQDN of broker node>:/tmp/cert-file .
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:test1234
At this point i now have a ca-cert.srl file and a cert-signed file on the Edge/CA node. But the ca-cert (certificate for the CA) + cert-signed (that I just generated) need to be on the broker nodes so I can import them into the individual broker's keystores, so I am going to copy them back there:
On all 3 brokers (1 node at a time):
scp <username>@<FQDN of Edge/CA node>:/tmp/ca-cert .
scp <username>@<FQDN of Edge/CA node>:/tmp/cert-signed .
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
From checking the directions, it appears that each broker should have a local file named "server.truststore.jks". But, I don't. Perhaps I followed something too literally? I did create a server.truststore.jks file in Step 2, but it's sitting on the Edge/CA node. This doesn't seem right, so I am going to instead execute the below command (from Step 2) on all 3 brokers. The ca-cert file was already copied to the brokers so this should work.
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
I then add the following lines to my server.properties file on each broker:
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.endpoint.identification.algorithm=HTTPS
ssl.key.password=test1234
ssl.keystore.location=/var/private/ssl/server.keystore.jks
ssl.keystore.password=test1234
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=test1234
listeners=SSL://<FQDN>:9093
security.inter.broker.protocol=SSL
When I start up Kafka after all of the changes above, I don't see any errors in the logs...but I don't see very many logging entries at all. The Kafka process is definitely up on all 3 brokers, but there's just not much in the logs. When I run the suggested:
openssl s_client -debug -connect FQDN:9093 -tls1
I get something of the form:
<hexdump>
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
...
Verify return code: 19 (self signed certificate in certificate chain)
I'm not sure if that "self signed certificate" error means I did something wrong above?
When I create the following properties file for my consumer/producer on my Edge/CA node:
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/client.truststore.jks
ssl.truststore.password=test1234
and start up a producer/consumer quick test:
kafka-console-consumer.sh --bootstrap-server broker1:9093,broker2:9093,broker3:9093 --topic withssl --consumer.config ssl.properties
kafka-console-producer.sh --broker-list broker1:9093,broker2:9093,broker3:9093 --topic withssl --producer.config ssl.properties
Both give me the same error:
[2017-08-30 18:07:58,233] WARN Bootstrap broker broker0:9093 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-30 18:07:58,544] WARN Bootstrap broker broker1:9093 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-08-30 18:07:58,760] WARN Bootstrap broker broker2:9093 disconnected (org.apache.kafka.clients.NetworkClient)
I'm not sure if my 3 brokers are set up incorrectly, or it my clients just can't connect. Does anyone have any advice?
