https://community.cloudera.com/t5/Archives-of-Support-Questions/Spark-shell-throwing-an-error-while-connecting-yarn-master/m-p/194787#M83831 <P>Exception in thread "main" java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:288) at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:248) at org.apache.spark.deploy.SparkSubmitArguments.&lt;init&gt;(SparkSubmitArguments.scala:120) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:130) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [spark@ip-10-0-10-76 ~]$</P> Wed, 26 Sep 2018 12:56:51 GMT subhash_parise3 2018-09-26T12:56:51Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Spark-shell-throwing-an-error-while-connecting-yarn-master/m-p/194787#M83831 <P>Exception in thread "main" java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:288) at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:248) at org.apache.spark.deploy.SparkSubmitArguments.&lt;init&gt;(SparkSubmitArguments.scala:120) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:130) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [spark@ip-10-0-10-76 ~]$</P> Wed, 26 Sep 2018 12:56:51 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Spark-shell-throwing-an-error-while-connecting-yarn-master/m-p/194787#M83831 subhash_parise3 2018-09-26T12:56:51Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Spark-shell-throwing-an-error-while-connecting-yarn-master/m-p/194788#M83832 <P>The above error wiped out once we have added HADOOP_CONF_DIR and YARN_CONF_DIR in .bashrc file in users home directory</P> Wed, 26 Sep 2018 14:04:38 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Spark-shell-throwing-an-error-while-connecting-yarn-master/m-p/194788#M83832 subhash_parise3 2018-09-26T14:04:38Z