We are trying to execute a spark structured streaming job that reads from Kafka and writes to HDFS. Spark on Yarn authentication is from one Kerberos server and Kafka is in another cluster authenticated by a different kerberos server.
In cluster mode, the same spark-submit works fine however in client mode it fails with the below error.
Command ./spark-submit --verbose --num-executors 4 --master yarn --deploy-mode client --driver-cores 2 --executor-cores 2 --executor-memory 1g --driver-memory 1g --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kf_client_jaas1.conf -Djava.security.krb5.realm=hdp261.xxxxx.com -Djava.security.krb5.kdc=172.40.9.101 -Dsun.security.krb5.debug=true" --files "/root/work/kf_client_jaas1.conf,/root/work/demo-spark/spark-hdp261/direp_lab.jks" --conf "spark.hadoop.yarn.client.failover-proxy-provider=org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider" --class org.xxxxx.spark.streaming.transform.KrbDebug "/root/work/spark-2.4.0-bin-hadoop2.7/jars/sparkstreaming-xxxxx-2.4.0_poc.jar"
ERROR
Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:940) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:587) at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60) at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:103) at org.apache.kafka.common.security.authenticator.LoginManager.(LoginManager.java:62) at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105) at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:147)
I am doing a kinit before the spark submit for the spark on yarn authentication.Can someone help in understanding the issue with this execution?
