Hello R,     Following suggests that HiveMetaStore isn't allowing connections from PRINCIPLE @ REALM.   Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: hive/ FQDN @ REALM is not allowed to impersonate PRINCIPLE @ REALM A very common reason for this to occur is configuring "hadoop.proxyuser.hive.groups" per steps from link [1]. If this access is required for the user running Spark jobs, you can add the group of this user to hadoop.proxyuser.hive.groups or the username of this user to hadoop.proxyuser.hive.users     [1] https://www.cloudera.com/documentation/enterprise/5-15-x/topics/sg_sentry_service_config.html#concept_z5b_42s_p4__section_lvc_4g4_rp ... View more

Hello Vijay,   Please see [1]. This use case isn't supported.   However, shared error suggests that executor isn't able to connect to Impala daemon due to authenitcation issues. This is because executor is running in a separate JVM and should acquire Kerberos TGT as well.   In order to perform this, you could make use of jaas configuration, see [2] and search for "To set up the JAAS login configuration file" (page 15). Once you have a tested Jaas login configuration and a keytab file, you could pass it as follows to the executors.     --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false" \ --conf "spark.yarn.dist.files=<path_to_keytab>.keytab,<path_to_keytab>/jaas.conf"   Alternatively, if your Impala can authenticate using LDAP, you could also test using it.   Hope this helps!   Thanks, Sudarshan   [1] https://www.cloudera.com/documentation/spark2/latest/topics/spark2_known_issues.html#ki_jdbc_datasource_api [2] https://www.cloudera.com/documentation/other/connectors/impala-jdbc/latest/Cloudera-JDBC-Driver-for-Impala-Install-Guide.pdf ... View more