Connect Storm to secured Kafka with Kafka Spout

[#1]

In the Hortonworks article about connecting the Kafka spout to a secure Kafka cluster it says that you need to set 'spoutConfig.securityProtocol=PLAINTEXTSASL' for the Kafka spout config (Configuring KafkaSpout to Connect to a Secure Kafka Cluster).

However the securityProtocol property does not exist as part of the config, so this article appears to be out of date.

KafkaConfig.java

In the Storm issue tracker they make mention of the fact that securityProtocol was never meant to be part of the public release (STORM-822: Kafka Spout New Consumer API #1131).

"securityProtocol is not needed any more, and was actually never in open source"

So my question is:

How can I correctly configure the Kafka Spout to connect to a secured Kafka cluster?

[#2]

@Ryan Spring

Please use the dependencies mentioned here : https://community.hortonworks.com/questions/27966/kafkaspout-fails-with-zookeeper-socket-issues-in-k...

securityProtocol property is available in hortonworks repo.

[#3 · Accepted solution]

@Ryan Spring

As part of Kafka Security work, we shipped Kerberos security in old consumer api's for Kafka. This is not part of Apache Kafka and hence Apache Storm's storm-kafka (old implementation) which uses old consumer API won't be able to talk to secure Apache Kafka.

This is only possible with HDP Kafka as we support old Kafka client API connecting Kerberos Kafka. So if you make dependency from the hortonworks artifacts you'll get this functionality.