Support Questions
Find answers, ask questions, and share your expertise

Launching a Spark-Submit job under Kerberos for Kafka

Through tinkering I have been able to partially launch a spark submit job using the following command, however soon after starting it crashes and gives me the exception outlined below:

Spark-Submit Command:
su spark -c 'export SPARK_MAJOR_VERSION=2; spark-submit \
--verbose \
--master yarn \
--driver-cores 5 \
--num-executors 3 --executor-cores 6 \
--principal spark@test.com \
--keytab /etc/security/keytabs/spark.headless.keytab \
--driver-java-options "-Djava.security.auth.login.config=kafka_client_jaas.conf"\
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \
--files "/tmp/kafka_client_jaas.conf,/tmp/kafka.service.keytab" \
--class au.com.XXX.XXX.spark.test.test test.jar application.properties'
EXCEPTION: 

Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not logn:the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user

WARN KerberosLogin: [Principal=kafka/test.com@test.com]: TGT renewal thread has been interrupted and will exit.

How can I get Kerberos to KINIT two principals at the same time? I'm assuming that is the problem here? I have tried adding another set of --principal/--keytab to the initial command, although this presented more permission issues within HDFS.

1 REPLY 1

@L V,

Can you adding/modifying this config to your kafka_client_jaas.conf and try running the job

useTicketCache=true

Do a kinit before running the spark submit command

Thanks,

Aditya

; ;