Is reading Kafka SASL_SSL from Spark streaming supported in HDP 2.4?

[#1]

I'm trying to use Spark streaming to read from a Kafka topic that is Kerberos authenticated and SSL (so, TLS) encrypted. The Kafka broker is in SASL_SSL mode. NiFi can read and write to these topics fine. My Spark streaming app can read fine if it's SASL_PLAINTEXT (using the HWX spark jars), but I'd like to know if SASL_SSL is possible. There are pieces of answers (many conflicting) to this question spread over the web (the most relevant is this JIRA saying it's not possible until Spark 2.0: https://issues.apache.org/jira/browse/SPARK-12177), but I'd like a definitive answer for the following stack versions:

HDP 2.4.2 so Spark 1.6.1

HDF 2.0 so Kafka 0.10.0.1

Java 8

Any tips, links to JIRAs, or best of all writeups of attempts would be appreciated!

[#2 · Accepted solution]

@Oliver Meyn

This is the correct jira https://issues.apache.org/jira/browse/SPARK-12177 and yes, SASL_SSL is only available from Spark 2.0 and not in HDP2.4.2 which has Spark 1.6.1.

[#3]

Here's another link that makes it clear (from the Apache Spark page): http://spark.apache.org/docs/latest/streaming-kafka-integration.html

[#4]

Great thanks for sharing..