Kerberos Issue

[#1]

Log on driver :

2021/02/18 18:10:09.612 DEBUG UserGroupInformation: [] hadoop login
2021/02/18 18:10:09.614 DEBUG UserGroupInformation: [] hadoop login commit
2021/02/18 18:10:09.617 DEBUG UserGroupInformation: [] using local user:UnixPrincipal: tss
2021/02/18 18:10:09.618 DEBUG UserGroupInformation: [] Using user: "UnixPrincipal: tss" with name tss
2021/02/18 18:10:09.618 DEBUG UserGroupInformation: [] User entry: "tss"
2021/02/18 18:10:09.622 DEBUG UserGroupInformation: [] Reading credentials from location set in HADOOP_TOKEN_FILE_LOCATION: /srv/BigData/hadoop/data1/nm/localdir/usercache/tss/appcache/application_1611987488531_0743/container_e13_1611987488531_0743_02_000001/container_tokens
2021/02/18 18:10:09.635 DEBUG UserGroupInformation: [] Loaded 2 tokens
2021/02/18 18:10:09.636 DEBUG UserGroupInformation: [] UGI loginUser:tss (auth:SIMPLE)
2021/02/18 18:10:09.637 INFO AMCredentialRenewer: [] Attempting to login to KDC using principal: tss@HADOOP.COM
2021/02/18 18:10:09.727 DEBUG UserGroupInformation: [] hadoop login
2021/02/18 18:10:09.728 DEBUG UserGroupInformation: [] hadoop login commit
2021/02/18 18:10:09.728 DEBUG UserGroupInformation: [] using kerberos user:tss@HADOOP.COM
2021/02/18 18:10:09.729 DEBUG UserGroupInformation: [] Using user: "tss@HADOOP.COM" with name tss@HADOOP.COM
2021/02/18 18:10:09.729 DEBUG UserGroupInformation: [] User entry: "tss@HADOOP.COM"
2021/02/18 18:10:09.730 INFO AMCredentialRenewer: [] Successfully logged into KDC.
2021/02/18 18:10:09.733 DEBUG UserGroupInformation: [] PrivilegedAction as:tss@HADOOP.COM (auth:KERBEROS)

Log on executor:

2021/02/18 18:10:19.426 DEBUG UserGroupInformation: [] hadoop login
2021/02/18 18:10:19.427 DEBUG UserGroupInformation: [] hadoop login commit
2021/02/18 18:10:19.428 DEBUG UserGroupInformation: [] using local user:UnixPrincipal: tss
2021/02/18 18:10:19.429 DEBUG UserGroupInformation: [] Using user: "UnixPrincipal: tss" with name tss
2021/02/18 18:10:19.429 DEBUG UserGroupInformation: [] User entry: "tss"
2021/02/18 18:10:19.432 DEBUG UserGroupInformation: [] Reading credentials from location set in HADOOP_TOKEN_FILE_LOCATION: /srv/BigData/hadoop/data1/nm/localdir/usercache/tss/appcache/application_1611987488531_0743/container_e13_1611987488531_0743_02_000002/container_tokens
2021/02/18 18:10:19.448 DEBUG UserGroupInformation: [] Loaded 2 tokens
2021/02/18 18:10:19.449 DEBUG UserGroupInformation: [] UGI loginUser:tss (auth:SIMPLE)
2021/02/18 18:10:19.449 DEBUG UserGroupInformation: [] PrivilegedAction as:tss (auth:SIMPLE) from:org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)

In driver ,am would renew the token ,ugi can login as kerberos. While in executor ,the ugi login simple that ugi
method hasKerberosCredentials() return a false. As as result , a exception has been throwed which reports "Could not locate Kerberos Principal on currently logged in user."

spark-submit --class cn.zwy.SparkWordCount \
--master yarn --deploy-mode cluster --conf 'spark.es.nodes=fusioninsight02:24100' \
--jars ~/elasticsearch-hadoop-7.10.2.jar \
--conf 'spark.es.ssl.enabled=true' \
--conf 'spark.es.net.ssl=true' \
--conf 'spark.es.resource=22lastk_eoi_2020_02_05' \
--conf 'es.net.ssl=true' \
--conf 'spark.es.security.authentication=kerberos' \
--conf 'es.security.authentication=kerberos' \
--conf 'spark.es.net.spnego.auth.elasticsearch.principal=elasticsearch/hadoop.hadoop.com@HADOOP.COM' \
--conf spark.yarn.submit.waitAppCompletion=false \
--conf spark.driver.extraJavaOptions="-Djava.security.krb5.conf=/opt/huawei/Bigdata/FusionInsight_BASE_6.5.1/3_49_KerberosClient/etc/kdc.conf" \
--conf spark.executor.extraJavaOptions="-Djava.security.krb5.conf=/opt/huawei/Bigdata/FusionInsight_BASE_6.5.1/3_49_KerberosClient/etc/kdc.conf" \
--conf spark.yarn.credentials.file=/tmp \
--principal tss@HADOOP.COM \
--keytab /data01/jax1.0/jax/user.keytab \
--conf 'hadoop.security.authentication=kerberos' \
~/spark-test-1.0-SNAPSHOT.jar

Anyone can help resolve this issue

[#2]

@Kolli 

Based on the logs and the Spark-submit command provided, it seems like there are discrepancies between the authentication mechanisms used in the driver and the executor environments, leading to authentication errors. Here are some potential issues and solutions:

1. Mismatch in Authentication Mechanisms:

   • The driver seems to authenticate using Kerberos (kerberos), while the executor uses simple authentication (SIMPLE).
   • Ensure consistency in the authentication mechanisms across the driver and executor environments.

2. Kerberos Configuration:

   • Verify that the Kerberos configuration (krb5.conf) provided in the spark.driver.extraJavaOptions and spark.executor.extraJavaOptions is correct and accessible by both the driver and executor.
   • Check if the Kerberos principal and keytab specified in the spark-submit command are accurate and valid.

3. SPNEGO Configuration:

   • Ensure that SPNEGO authentication is properly configured for the Spark Elasticsearch connector.
   • Verify that the SPNEGO principal (elasticsearch/hadoop.hadoop.com@HADOOP.COM) specified in the spark-submit command matches the one configured in the environment.

4. Permission Issues:

   • Check the permissions of the keytab file (user.keytab) specified in the spark-submit command to ensure that it is accessible by both the driver and executor.

5. Token Renewal:

   • Review the token renewal mechanism to ensure that tokens are properly renewed and propagated to the executor.

To address the issue, consider the following steps:

• Ensure that both the driver and executor environments are configured consistently for Kerberos authentication.
• Double-check all Kerberos-related configurations, including the Kerberos principal, keytab, and krb5.conf file paths.
• Verify that the SPNEGO authentication settings are correctly configured for the Spark Elasticsearch connector.
• Check for any permission issues with the keytab file or other Kerberos-related files.
• Review the token renewal mechanism to ensure proper token propagation.

Regards,

Chethan YM