Support Questions
Find answers, ask questions, and share your expertise

Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

New Contributor

Hi everyone

I'm currently using HDP 2.5 (Spark 1.6 and Kafka 0.10) and I'd like to know if Spark Streaming 1.6 supports Kafka 0.10 with Kerberos.

I've read contradictory information so far. Some articles state that you need Spark 2.0 to use a Kerberized broker with Kafka 0.10. Some of the others state that it's possible but dont really indicate how to achieve that.

Can you help?

Thank you

7 REPLIES 7

Re: Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

It worked for me with HDP 2.4.2 and the Kafka DirectStreaming API, so I expect it to still work with HDP 2.5

You need to add the security protocol into your Spark source code as Kafka Parameter

var kafkaParams = Map("metadata.broker.list" -> server,
                      "group.id"             -> groupIid)
if (useKerberos) kafkaParams = kafkaParams + ("security.protocol" ->"PLAINTEXTSASL")
val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder]
                                               (ssc, kafkaParams, Set(topic))

Then create a spark.jaas file for accessing Kafka and Zookeeper

KafkaClient {
 com.sun.security.auth.module.Krb5LoginModule required
 doNotPrompt=true
 useTicketCache=true
 principal="streaming@EXAMPLE.COM"
 useKeyTab=true
 serviceName="kafka"
 keyTab="streaming-kafka.keytab"
 client=true;
};
Client {
 com.sun.security.auth.module.Krb5LoginModule required
 doNotPrompt=true
 useTicketCache=true
 principal="streaming@EXAMPLE.COM"
 useKeyTab=true
 serviceName="zookeeper"
 keyTab="streaming-kafka.keytab"
 client=true;
};

and call Spark like

spark-submit --master=yarn --deploy-mode=client \
    --files "spark.jaas,streaming-kafka.keytab" \
    --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=spark.jaas" \
    --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=spark.jaas" \
    --class streaming.MyStreamingClass /path/to/my-streaming-assembly-0.1.jar

Ensure that keytab exist and keytab user can access Kafka and Zookeeper.

Re: Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

New Contributor

Hi Bernard

Thanks a lot for your detailed answer. Can you confirm that you're using the same version for Spark (1.6) and Kafka (0.10)

Cheers

Re: Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

New Contributor

Anyone know if we can make Spark Streaming 1.6 with Kafka 0.9 with Kerberos authenticaiton?

Re: Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

I'm using Spark streaming 1.6.2 and reading from a kerberized Kafka 0.9. In this URL I posted an example Spark application, how to configure the jaas file and how to start the application.

Re: Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

@Cyril Doe Support for Spark Streaming and Kafka in a Secure Cluster (Kerberos-enabled) is supported from HDP 2.4.2 and above.

Here is the documentation for same : http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_spark-component-guide/content/using-spark...

Re: Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

@john huang I am using spark 1.6.1 and kafka 0.9 on HDP which is kerberosed.

Re: Does Spark Streaming 1.6 support Kafka 0.10 with Kerberos authentication

Expert Contributor

Apache spark does not support Kafka 0.9 or 0.10 in the Spark 1.6.x. Apache Spark 1.6 is not accepting new features like this. Only bug fixes.

Inside HDP from 2.4.2 onwards, Spark supports the corresponding Kafka version in the same HDP. So if Kafka in HDP is 0.9 then Spark in that HDP will support secure Kerberos for 0.9. If Kafka in HDP is 0.10 then Spark in the HDP will support Kerberos access for Kafka 0.10.

How to set that up is documented here.

If you want to use Kafka 0.10 native support with SSL also, then you can try using the package here.