Support Questions
Find answers, ask questions, and share your expertise

How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

Explorer

Specifications:

  • HDP 2.3.2
  • Kerberos enabled
  • Kafka topic exists and user <username> has read access
  • Kafka topic is readable/writable using the Kafka command line tools with specified user
  • We already have a Spark streaming application that works fine in an unsecure cluster reading from a Kafka topic.

What would be a working example of a Spark streaming job that reads input from Kafka in a secure cluster under above conditions?

We made a Spark streaming job work that reads/writes into secure HBase and thought it couldn't be that different to do it with Kafka.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

spark streaming haven't yet enabled security for their kafka connector

View solution in original post

21 REPLIES 21

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

spark streaming haven't yet enabled security for their kafka connector

View solution in original post

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

@schintalapani - Do you have more details? Such as if there are JIRA issues or dev email threads tracking this?

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

is not enable for the DirectApi connector, but is enable with the old one.

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

New Contributor

Can you shed some light on how you use Spark Streaming to read/write to HBase? I have hard time to use Spark/Scala to access HBase. Thanks!

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

Expert Contributor

I have made a small change to the spark streaming code here : https://github.com/davidtambjss/spark-release/tree/HDP-2.3.0.0-KERBEROS_KAFKA_STREAMING and was able to get Spark streaming to work with kafka with kerberos.

All you need rebuild spark-streaming-kafka_2.10-1.3.1.2.3.0.0-2557.jar from there and use that jar.

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

New Contributor

Hi David,

Can you please provide the running code.

Inside pom.xml im getting below compilation error:

Project build error: Non-resolvable parent POM for com.accenture.ngap:spark-test:[unknown-version]: Failure to transfer

Thanks,

Krishna

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

Checked with engineering today and the feature will be officially landing in upcoming HDP 2.4.2 patch. There will be documentation around the fact that users will need to use HDP spark streaming kafka jars (instead of vanilla Apache)

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

Expert Contributor

@Ali Bajwa we have HDP 2.4.2 and when we try to consume the messages form the Secured Kafka topics using Spark Streaming (spark 1.6.1) we can't consume any messages.

I followed the documentation on https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/spark-streaming-kafk...

Was this patch after 2.4.2 or am I missing something.

Thanks.