After a hard day our team solved the issue and we have created a little
Spark application that can read from a kerberized Kafka and launching using yarn-cluster mode. Attached you have the pom.xml needed (very important use the Kafka 0.9 dependency) and the java code.
It is very important to
use the old Kafka API and to use PLAINTEXTSASL (and not SASL_PLAINTEXT, a value coming from the new Kafka API) when configuring the Kafka Stream. The jaas.conf file is the same that @Alexander Bij shared. Be careful because the keytab path is not a local path, is the keytab where the executor store the file (by default you only have to indicated the name of the keytab). KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=false
useKeyTab=true
principal="serviceaccount@DOMAIN.COM"
keyTab="serviceaccount.headless.keytab"
renewTicket=true
storeKey=true
serviceName="kafka";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=false
useKeyTab=true
principal="serviceaccount@DOMAIN.COM"
keyTab="serviceaccount.headless.keytab"
renewTicket=true
storeKey=true
serviceName="zookeeper";
};
Kafka will use this information to retrieve the kerberos configuration required when connecting to ZK and to the broker. In order to do that, the executor will require the keytab file to get the kerberos token before connecting to the broker. That is the reason of sending also the keytab to the container when launching the application. Please, notice that the extraJavaOptions refers to the container local path (the file will be placed in the root folder of the temp configuration) but the file path will require the local path where you have the jaas file in the server that is starting up the application. spark-submit (all your stuff) \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \
--files "your_other_files,kafka_client_jaas.conf,serviceaccount.headless.keytab" \
(rest of your stuff)
Also verify that the parameter where you indicate the jaas file is java.security.auth.login.config. And that's all, following above instruction and using the attached code you will be able to read from kerberized Kafka using Spark 1.5.2. In our Kerberized environment we have Ambari 2.2.0.0, HDP 2.3.4.0 and Spark 1.5.2. I hope this information help you.
... View more
