question Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster? in Archives of Support Questions (Read Only)

How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

condla — Mon, 14 Dec 2015 18:54:38 GMT

Specifications:

HDP 2.3.2
Kerberos enabled
Kafka topic exists and user <username> has read access
Kafka topic is readable/writable using the Kafka command line tools with specified user
We already have a Spark streaming application that works fine in an unsecure cluster reading from a Kafka topic.

What would be a working example of a Spark streaming job that reads input from Kafka in a secure cluster under above conditions?

We made a Spark streaming job work that reads/writes into secure HBase and thought it couldn't be that different to do it with Kafka.

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

schintalapani — Wed, 16 Dec 2015 03:13:01 GMT

spark streaming haven't yet enabled security for their kafka connector

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

sroberts — Fri, 18 Dec 2015 21:18:06 GMT

@schintalapani - Do you have more details? Such as if there are JIRA issues or dev email threads tracking this?

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

wuz — Sat, 23 Jan 2016 13:39:43 GMT

Can you shed some light on how you use Spark Streaming to read/write to HBase? I have hard time to use Spark/Scala to access HBase. Thanks!

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

David_Tam — Tue, 26 Jan 2016 21:30:27 GMT

I have made a small change to the spark streaming code here : https://github.com/davidtambjss/spark-release/tree/HDP-2.3.0.0-KERBEROS_KAFKA_STREAMING and was able to get Spark streaming to work with kafka with kerberos.

All you need rebuild spark-streaming-kafka_2.10-1.3.1.2.3.0.0-2557.jar from there and use that jar.

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

nsabharwal — Tue, 26 Jan 2016 22:20:06 GMT

@schintalapani

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

abajwa — Thu, 21 Apr 2016 00:50:18 GMT

Checked with engineering today and the feature will be officially landing in upcoming HDP 2.4.2 patch. There will be documentation around the fact that users will need to use HDP spark streaming kafka jars (instead of vanilla Apache)

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

aromero — Thu, 21 Apr 2016 06:40:28 GMT

If you want to test it, even on HDP 2.3.x (Spark 1.5.2) you can use the following library which has been compiled with the necessary changes in:

https://github.com/beto983/Streaming-Kafka-Spark-1.5.2/blob/master/spark-streaming-kafka_2.10-1.5.2.jar

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

MrBee — Thu, 21 Apr 2016 14:38:43 GMT

Hi Stefan Kupstaitis-Dunkler,

We are using HDP-2.3.4.0 and use Kafka en SparkStreaming (Scala & Python) on a (Kerberos + Ranger) secured Cluster.

You need to add a jaas config location to the spark-sumbit command. We are using it in yarn-client mode. The kafka_client_jaas.conf file is send as a resource with the --files option and available in the yarn-container. We did not get ticket renewal working yet...

spark-submit (all your stuff) \
  --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.conf=kafka_client_jaas.conf" \
  --files "your_other_files,kafa_client_jaas.conf,serviceaccount.headless.keytab" \
  (rest of your stuff)

# --principal and --keytab does not work and conflict with --files keytab.
# The jaas file will be placed in the yarn-containers by Spark.
# The file contains the reference to the keytab-file and the principal for Kafka and ZooKeeper:

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=false
  useKeyTab=true
  principal="serviceaccount@DOMAIN.COM"
  keyTab="serviceaccount.headless.keytab"
  renewTicket=true
  storeKey=true
  serviceName="kafka";
};
Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=false
  useKeyTab=true
  principal="serviceaccount@DOMAIN.COM"
  keyTab="serviceaccount.headless.keytab"
  renewTicket=true
  storeKey=true
  serviceName="zookeeper";
}

If you need more info, feel free to ask.

Greetings,

Alexander

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

teixeira_javier — Thu, 28 Apr 2016 14:18:31 GMT

Do you have the code with the changes that you did in that jar?

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

teixeira_javier — Thu, 28 Apr 2016 14:20:13 GMT

I have tried this approach and doesn't work 😞 . Spark doesn't throw any error, but doesn't read anything from the kerberized Kafka.

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

teixeira_javier — Thu, 28 Apr 2016 19:33:19 GMT

is not enable for the DirectApi connector, but is enable with the old one.

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

teixeira_javier — Thu, 28 Apr 2016 20:32:02 GMT

After a hard day our team solved the issue and we have created a little Spark application that can read from a kerberized Kafka and launching using yarn-cluster mode. Attached you have the pom.xml needed (very important use the Kafka 0.9 dependency) and the java code.

It is very important to use the old Kafka API and to use PLAINTEXTSASL (and not SASL_PLAINTEXT, a value coming from the new Kafka API) when configuring the Kafka Stream.

The jaas.conf file is the same that @Alexander Bij shared. Be careful because the keytab path is not a local path, is the keytab where the executor store the file (by default you only have to indicated the name of the keytab).

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=false
  useKeyTab=true
  principal="serviceaccount@DOMAIN.COM"
  keyTab="serviceaccount.headless.keytab"
  renewTicket=true
  storeKey=true
  serviceName="kafka";
};
Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useTicketCache=false
  useKeyTab=true
  principal="serviceaccount@DOMAIN.COM"
  keyTab="serviceaccount.headless.keytab"
  renewTicket=true
  storeKey=true
  serviceName="zookeeper";
};

Kafka will use this information to retrieve the kerberos configuration required when connecting to ZK and to the broker. In order to do that, the executor will require the keytab file to get the kerberos token before connecting to the broker. That is the reason of sending also the keytab to the container when launching the application. Please, notice that the extraJavaOptions refers to the container local path (the file will be placed in the root folder of the temp configuration) but the file path will require the local path where you have the jaas file in the server that is starting up the application.

spark-submit (all your stuff) \
  --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \
  --files "your_other_files,kafka_client_jaas.conf,serviceaccount.headless.keytab" \
  (rest of your stuff)

Also verify that the parameter where you indicate the jaas file is java.security.auth.login.config.

And that's all, following above instruction and using the attached code you will be able to read from kerberized Kafka using Spark 1.5.2.

In our Kerberized environment we have Ambari 2.2.0.0, HDP 2.3.4.0 and Spark 1.5.2. I hope this information help you.

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

Pradeep_misra — Sat, 14 May 2016 04:36:06 GMT

Are there examples for HDP 2.3.2 - Spark Streaming (1.4.1) and Kafka(0.8.2) with java and kerberos environment

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

revanthkumar95 — Thu, 18 Aug 2016 16:34:17 GMT

HI Team ,

I am currently using Spark Streaming with Kafka without any SSL , it works fine .

But we need to enable SSL for Spark Streaming with Kafka.

Is this supported ?

I see spark 2.0 also supports kafka 0.8 version, but SSL is added in 0.9 kafka version .

Is there any way through which HDP is providing SSL enabled spark streaming with kafka ?

If yes . can any one share the document link or any info ?

Thanks

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

revanthkumar95 — Thu, 18 Aug 2016 16:36:47 GMT

Hi Stefan,

Spark Streaming +Kafka +SSL

Did you resolve this issue or got any solution for this ?

can u please share if ...

Thanks

Revanth

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

smartninja723 — Mon, 03 Oct 2016 20:50:26 GMT

@Ali Bajwa we have HDP 2.4.2 and when we try to consume the messages form the Secured Kafka topics using Spark Streaming (spark 1.6.1) we can't consume any messages.

I followed the documentation on https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/spark-streaming-kafka-kerb.html

Was this patch after 2.4.2 or am I missing something.

Thanks.

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

smartninja723 — Mon, 03 Oct 2016 20:51:03 GMT

I think we also have same problem with Kafka 0.9, Spark 1.6.1 and HDP 2.4.2

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

bikas — Thu, 06 Oct 2016 10:18:19 GMT

What is the issue you are facing after following the above documentation in HDP 2.4.2?

Re: How to read from a Kafka topic using Spark (streaming) in a secure Cluster?

teixeira_javier — Fri, 16 Dec 2016 19:54:44 GMT

Did you try with the code that I shared on https://community.hortonworks.com/comments/30193/view.html some comments above?