https://community.cloudera.com/t5/Support-Questions/quot-bad-substitution-quot-error-running-Spark-on-Yarn/m-p/135406#M98065 <P>Using HDP 2.5, submitting from a vanilla Spark 2.1.0 (i.e., not HDP), and deploying with deploy-mode "cluster", we were successful by using a variation on the suggestions of <A rel="user" href="https://community.cloudera.com/users/132/abajwa.html" nodeid="132">@Ali Bajwa</A>:</P><UL><LI>Creating a file named "java-opts" in our Spark conf directory containing "-Dhdp.version=2.5.x.x-xx" (subsituting our specific version)</LI><LI>Adding into our Spark configuration (via the Spark submit --conf option) "spark.driver.extraJavaOptions=-Dhdp.version=2.5.x.x-xx" and "spark.executor.extraJavaOptions=-Dhdp.version=2.5.x.x-xx" </LI></UL><P>Both of the creation of the "java-opts" file and the spark configuration modifications were required for success in our case.</P><P>The "spark.driver.extraJavaOptions" option was definitely necessary in our case, but the "spark.executor.extraJavaOptions" may not be necessary. As I understand it, the "spark.yarn.am.extraJavaOptions" option that Ali mentioned is not relevant in cluster mode.</P><P><A rel="user" href="https://community.cloudera.com/users/132/abajwa.html" nodeid="132"></A> </P><P><A rel="user" href="https://community.cloudera.com/users/132/abajwa.html" nodeid="132"></A> </P> Wed, 14 Feb 2018 02:20:52 GMT mathis_andrew 2018-02-14T02:20:52Z