How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

[#1]

Specifications:

• HDP 2.3.2
• Kerberos enabled
• Kafka topic exists and user <username> has read access
• Kafka topic is readable/writable using the Kafka command line tools with specified user
• We already have a Spark streaming application that works fine in an unsecure cluster reading from a Kafka topic.

What would be a working example of a Spark streaming job that reads input from Kafka in a secure cluster under above conditions?

We made a Spark streaming job work that reads/writes into secure HBase and thought it couldn't be that different to do it with Kafka.

[#2]

I think we also have same problem with Kafka 0.9, Spark 1.6.1 and HDP 2.4.2

[#3]

Hi Stefan,

Spark Streaming +Kafka +SSL

Did you resolve this issue or got any solution for this ?

can u please share if ...

Thanks

Revanth