Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark-shell throwing an error while connecting yarn master mode in HDP 3.0?

Solved Go to solution
Highlighted

Spark-shell throwing an error while connecting yarn master mode in HDP 3.0?

Super Collaborator

Exception in thread "main" java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:288) at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:248) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:120) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:130) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [spark@ip-10-0-10-76 ~]$

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Spark-shell throwing an error while connecting yarn master mode in HDP 3.0?

Super Collaborator

The above error wiped out once we have added HADOOP_CONF_DIR and YARN_CONF_DIR in .bashrc file in users home directory

1 REPLY 1

Re: Spark-shell throwing an error while connecting yarn master mode in HDP 3.0?

Super Collaborator

The above error wiped out once we have added HADOOP_CONF_DIR and YARN_CONF_DIR in .bashrc file in users home directory

Don't have an account?
Coming from Hortonworks? Activate your account here