Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Connect Storm to secured Kafka with Kafka Spout

Solved Go to solution
Highlighted

Connect Storm to secured Kafka with Kafka Spout

New Contributor

In the Hortonworks article about connecting the Kafka spout to a secure Kafka cluster it says that you need to set 'spoutConfig.securityProtocol=PLAINTEXTSASL' for the Kafka spout config (Configuring KafkaSpout to Connect to a Secure Kafka Cluster).

However the securityProtocol property does not exist as part of the config, so this article appears to be out of date.

KafkaConfig.java

In the Storm issue tracker they make mention of the fact that securityProtocol was never meant to be part of the public release (STORM-822: Kafka Spout New Consumer API #1131).

"securityProtocol is not needed any more, and was actually never in open source"

So my question is:

How can I correctly configure the Kafka Spout to connect to a secured Kafka cluster?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Connect Storm to secured Kafka with Kafka Spout

@Ryan Spring

As part of Kafka Security work, we shipped Kerberos security in old consumer api's for Kafka. This is not part of Apache Kafka and hence Apache Storm's storm-kafka (old implementation) which uses old consumer API won't be able to talk to secure Apache Kafka.

This is only possible with HDP Kafka as we support old Kafka client API connecting Kerberos Kafka. So if you make dependency from the hortonworks artifacts you'll get this functionality.

View solution in original post

2 REPLIES 2
Highlighted

Re: Connect Storm to secured Kafka with Kafka Spout

@Ryan Spring

Please use the dependencies mentioned here : https://community.hortonworks.com/questions/27966/kafkaspout-fails-with-zookeeper-socket-issues-in-k...

securityProtocol property is available in hortonworks repo.

Highlighted

Re: Connect Storm to secured Kafka with Kafka Spout

@Ryan Spring

As part of Kafka Security work, we shipped Kerberos security in old consumer api's for Kafka. This is not part of Apache Kafka and hence Apache Storm's storm-kafka (old implementation) which uses old consumer API won't be able to talk to secure Apache Kafka.

This is only possible with HDP Kafka as we support old Kafka client API connecting Kerberos Kafka. So if you make dependency from the hortonworks artifacts you'll get this functionality.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here