Support Questions

Find answers, ask questions, and share your expertise

How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

avatar
Contributor

Specifications:

  • HDP 2.3.2
  • Kerberos enabled
  • Kafka topic exists and user <username> has read access
  • Kafka topic is readable/writable using the Kafka command line tools with specified user
  • We already have a Spark streaming application that works fine in an unsecure cluster reading from a Kafka topic.

What would be a working example of a Spark streaming job that reads input from Kafka in a secure cluster under above conditions?

We made a Spark streaming job work that reads/writes into secure HBase and thought it couldn't be that different to do it with Kafka.

1 ACCEPTED SOLUTION

avatar

spark streaming haven't yet enabled security for their kafka connector

View solution in original post

21 REPLIES 21

avatar
Expert Contributor

I think we also have same problem with Kafka 0.9, Spark 1.6.1 and HDP 2.4.2

avatar
New Contributor

Hi Stefan,

Spark Streaming +Kafka +SSL

Did you resolve this issue or got any solution for this ?

can u please share if ...

Thanks

Revanth