Support Questions
Find answers, ask questions, and share your expertise

Launching a Spark-Submit job under Kerberos for Kafka

Through tinkering I have been able to partially launch a spark submit job using the following command, however soon after starting it crashes and gives me the exception outlined below:

Spark-Submit Command:
su spark -c 'export SPARK_MAJOR_VERSION=2; spark-submit \
--verbose \
--master yarn \
--driver-cores 5 \
--num-executors 3 --executor-cores 6 \
--principal \
--keytab /etc/security/keytabs/spark.headless.keytab \
--driver-java-options ""\
--conf "" \
--files "/tmp/kafka_client_jaas.conf,/tmp/kafka.service.keytab" \
--class test.jar'

Caused by: org.apache.kafka.common.KafkaException: Could not logn:the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user

WARN KerberosLogin: [Principal=kafka/]: TGT renewal thread has been interrupted and will exit.

How can I get Kerberos to KINIT two principals at the same time? I'm assuming that is the problem here? I have tried adding another set of --principal/--keytab to the initial command, although this presented more permission issues within HDFS.


@L V,

Can you adding/modifying this config to your kafka_client_jaas.conf and try running the job


Do a kinit before running the spark submit command



; ;