https://community.cloudera.com/t5/Support-Questions/SPARK-SUBMIT-issue-with-SPARK-2-2/m-p/217623#M179532 <P>We are on HDP 2.6.3 and using SPARK 2.2 and running the job using on YARN CLUSTER mode.</P><P>using spark-submit and the spark-env.sh contains <STRONG>SPARK_YARN_DIST_FILES="/etc/spark2/conf/hive-site.xml,/etc/spark2/conf/hbase-site.xml" </STRONG>but these values are not honored.</P><PRE>spark-submit --class com.virtuslab.sparksql.MainClass --master yarn --deploy-mode cluster /tmp/spark-hive-test/spark_sql_under_the_hood-spark2.2.0.jar</PRE><P>This is trying to connect to Hive and fetch the data from a table, but it fails with table on not found in database:</P><PRE> diagnostics: User class threw exception: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'xyz' not found in database 'qwerty'; ApplicationMaster host: 121.121.121.121 ApplicationMaster RPC port: 0 queue: default start time: 1523616607943 final status: FAILED tracking URL: <A href="https://managenode002xxserver:8090/proxy/application_1523374609937_10224/" target="_blank">https://managenode002xxserver:8090/proxy/application_1523374609937_10224/</A> user: abc123 Exception in thread "main" org.apache.spark.SparkException: Application application_1523374609937_10224 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1187) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1233) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:782) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)</PRE><P>The same works when we pass the --files parameter:</P><PRE>spark-submit --class com.virtuslab.sparksql.MainClass --master yarn --deploy-mode cluster --files /etc/spark2/conf/hive-site.xml /tmp/spark-hive-test/spark_sql_under_the_hood-spark2.2.0.jar</PRE><P>Result attached.</P><P>Any pointers why it is not using picking up <STRONG>SPARK_YARN_DIST_FILES</STRONG>?</P><P>Thanks</P><P>Venkat</P> Fri, 13 Apr 2018 18:01:19 GMT dheer_vijji_rag 2018-04-13T18:01:19Z