Getting errors on submitting a spark job

sahiljankaw — Sat, 06 Oct 2018 08:32:41 GMT

[spark@vm1 spark2-client]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 1 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples_2.11-2.3.1.3.0.1.0-187.jar 10

Logs

Traceback (most recent call last):
  File "/usr/bin/hdp-select", line 448, in <module>
    listPackages(getPackages("all"))
  File "/usr/bin/hdp-select", line 266, in listPackages
    os.path.basename(os.path.dirname(os.readlink(linkname))))

OSError: [Errno 22] Invalid argument: '/usr/hdp/current/oozie-client'

ls: cannot access /usr/hdp//hadoop/lib: No such file or directory

Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx
at org.apache.spark.launcher.Main.main(Main.java:118)

Re: Getting errors on submitting a spark job

sandyy006 — Sat, 06 Oct 2018 15:59:34 GMT

@Sahil Kaw

"/usr/hdp/current/oozie-client" should be a symlink of "/usr/hdp/<version>/oozie". Move/remove "/usr/hdp/current/oozie-client" directory out of "/usr/hdp" and refresh the oozie client from ambari and you should be good.

question Re: Getting errors on submitting a spark job in Archives of Support Questions (Read Only)

Getting errors on submitting a spark job

Re: Getting errors on submitting a spark job