question Re: Is reading Kafka SASL_SSL from Spark streaming supported in HDP 2.4? in Archives of Support Questions (Read Only)

Is reading Kafka SASL_SSL from Spark streaming supported in HDP 2.4?

oliver — Sat, 26 Nov 2016 03:36:46 GMT

I'm trying to use Spark streaming to read from a Kafka topic that is Kerberos authenticated and SSL (so, TLS) encrypted. The Kafka broker is in SASL_SSL mode. NiFi can read and write to these topics fine. My Spark streaming app can read fine if it's SASL_PLAINTEXT (using the HWX spark jars), but I'd like to know if SASL_SSL is possible. There are pieces of answers (many conflicting) to this question spread over the web (the most relevant is this JIRA saying it's not possible until Spark 2.0: https://issues.apache.org/jira/browse/SPARK-12177), but I'd like a definitive answer for the following stack versions:

HDP 2.4.2 so Spark 1.6.1

HDF 2.0 so Kafka 0.10.0.1

Java 8

Any tips, links to JIRAs, or best of all writeups of attempts would be appreciated!

Re: Is reading Kafka SASL_SSL from Spark streaming supported in HDP 2.4?

sandyy006 — Sat, 26 Nov 2016 04:30:38 GMT

@Oliver Meyn

This is the correct jira https://issues.apache.org/jira/browse/SPARK-12177 and yes, SASL_SSL is only available from Spark 2.0 and not in HDP2.4.2 which has Spark 1.6.1.

Re: Is reading Kafka SASL_SSL from Spark streaming supported in HDP 2.4?

oliver — Mon, 28 Nov 2016 23:22:20 GMT

Here's another link that makes it clear (from the Apache Spark page): http://spark.apache.org/docs/latest/streaming-kafka-integration.html

Re: Is reading Kafka SASL_SSL from Spark streaming supported in HDP 2.4?

sandyy006 — Tue, 29 Nov 2016 01:13:00 GMT

Great thanks for sharing..