question Re: Connect Storm to secured Kafka with Kafka Spout in Archives of Support Questions (Read Only)

Connect Storm to secured Kafka with Kafka Spout

ryan_spring — Mon, 29 Aug 2016 20:07:41 GMT

In the Hortonworks article about connecting the Kafka spout to a secure Kafka cluster it says that you need to set 'spoutConfig.securityProtocol=PLAINTEXTSASL' for the Kafka spout config (Configuring KafkaSpout to Connect to a Secure Kafka Cluster).

However the securityProtocol property does not exist as part of the config, so this article appears to be out of date.

KafkaConfig.java

In the Storm issue tracker they make mention of the fact that securityProtocol was never meant to be part of the public release (STORM-822: Kafka Spout New Consumer API #1131).

"securityProtocol is not needed any more, and was actually never in open source"

So my question is:

How can I correctly configure the Kafka Spout to connect to a secured Kafka cluster?

Re: Connect Storm to secured Kafka with Kafka Spout

sandyy006 — Mon, 29 Aug 2016 20:22:16 GMT

@Ryan Spring

Please use the dependencies mentioned here : https://community.hortonworks.com/questions/27966/kafkaspout-fails-with-zookeeper-socket-issues-in-k.html

securityProtocol property is available in hortonworks repo.

Re: Connect Storm to secured Kafka with Kafka Spout

schintalapani — Mon, 27 Feb 2017 02:30:42 GMT

@Ryan Spring

As part of Kafka Security work, we shipped Kerberos security in old consumer api's for Kafka. This is not part of Apache Kafka and hence Apache Storm's storm-kafka (old implementation) which uses old consumer API won't be able to talk to secure Apache Kafka.

This is only possible with HDP Kafka as we support old Kafka client API connecting Kerberos Kafka. So if you make dependency from the hortonworks artifacts you'll get this functionality.