Created on 03-10-2017 12:47 AM
Stunnel is a proxy that can make insecure network transmission secure by wrapping it with SSL. This article contains example and illustrations describing how it works and how to configure it.
Most part of it is derived from this informative Git comment I wouldn't be able to set it up without this comment. Thank you for sharing such detailed example.
Let's see how it can be applied to NiFi GetKafka. I used two servers for this experimentation. 0 and 1.server.aws.mine. A single Zookeeper and Kafka broker is running on 0.server. A GetKafka NiFi processor in 1.server consumes messages through Stunnel:
127.0.0.1:9092. If Zookeeper is in different server (recommended) and you need to secure this connection via Stunnel as well, then you need to apply the same method as the one used between GetKafka and Zookeeper.
Zookeeper Connection Stringis set to
127.0.0.1:2181which is local Stunnel is listening to. Then the local Stunnel on 1.server proxies the request to
0.server:2181over SSL. At 0.server, the request is proxied again by the Stunnel running on 0.server, then finally arrives at Zookeeper.
127.0.0.1:9092, GetKafka (Kafka client) sends request to
127.0.0.1:9092, and the request eventually transferred to the Broker through Stunnel pair.
Here is the relevant configurations in 1.server's stunnel.conf (entire file is available here):
client = yes [zookeeper] accept = 127.0.0.1:2181 connect = 0.server.aws.mine:2181 [kafka] accept = 127.0.0.1:9092 connect = 0.server.aws.mine:9092
And this is for 0.server (entire file is available here):
client = no [zookeeper] accept = 0.server.aws.mine:2181 connect = 127.0.0.1:2181 [kafka] accept = 0.server.aws.mine:9092 connect = 127.0.0.1:9092
Each Stunnel server has to have its own pem file containing a private key and a certificate. Also, a CA certificate file (or directory) is also needed to authorize client access.
nifi.propertiesfor each server. Server's key and cert can be extracted from keystore.jks. To do so, convert keystore.jks into keystore.p12 file by following commands (credit goes to this Stackoverflow) :
# It's not important which server to run the toolkit on. $ ./bin/tls-toolkit.sh standalone -n [0-1].server.aws.mine -C 'CN=server,OU=mine' # Password for keystore.jks can be found in generated nifi.properties 'nifi.security.keystorePasswd'. $ keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.p12 -srcstoretype jks -deststoretype pkcs12
$ openssl pkcs12 -in keystore.p12 -nokeys -out cert.pem $ openssl pkcs12 -in keystore.p12 -nodes -nocerts -out key.pem
$ cat key.pem cert.pem >> stunnel.pem
CAFilefor Stunnel on 0.server.
In stunnel.conf on 0.server, following settings are needed to enable client cert verification:
verify = 3 CAFile = /etc/stunnel/certs
Refer Stunnel manual for further description on these configurations.
I confirmed that GetKafka running on 1.server can consume messages through Stunnel. If I used a cert which is not configured in the certs file on 0.server, GetKafka got timeout exception as follows:
2017-03-09 06:50:48,690 WARN [Timer-Driven Process Thread-5] o.apache.nifi.processors.kafka.GetKafka GetKafka[id=b0a21b5d-015a-1000-fbba-2648095ae625] Executor did not stop in 30 sec. Terminated. 2017-03-09 06:50:48,691 WARN [Timer-Driven Process Thread-5] o.apache.nifi.processors.kafka.GetKafka GetKafka[id=b0a21b5d-015a-1000-fbba-2648095ae625] Timed out after 60000 milliseconds while waiting to get connection java.util.concurrent.TimeoutException: null at java.util.concurrent.FutureTask.get(FutureTask.java:205) [na:1.8.0_121] at org.apache.nifi.processors.kafka.GetKafka.onTrigger(GetKafka.java:348) ~[nifi-kafka-0-8-processors-1.1.2.jar:1.1.2] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.2.jar:1.1.2] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.2.jar:1.1.2]
# Install sudo yum -y install stunnel # Edit config sudo vi /etc/stunnel/stunnel.conf # Start sudo stunnel # Stop sudo kill `cat /var/run/stunnel.pid`
Although Stunnel works with GetKafka and Kafka 0.8.x, I recommend to use newer version of Kafka and ConsumeKafka NiFi processor with SSL if possible. As it's written in the Git comment, this workaround is not scalable (in terms of required administration tasks) and complicated.