- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Getting errors on submitting a spark job
- Labels:
-
Apache Hadoop
-
Apache Spark
Created ‎10-06-2018 01:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
[spark@vm1 spark2-client]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 1 --driver-memory 512m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples_2.11-2.3.1.3.0.1.0-187.jar 10
Logs Traceback (most recent call last): File "/usr/bin/hdp-select", line 448, in <module> listPackages(getPackages("all")) File "/usr/bin/hdp-select", line 266, in listPackages os.path.basename(os.path.dirname(os.readlink(linkname))))
OSError: [Errno 22] Invalid argument: '/usr/hdp/current/oozie-client'
ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx at org.apache.spark.launcher.Main.main(Main.java:118)
Created ‎10-06-2018 08:59 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
"/usr/hdp/current/oozie-client" should be a symlink of "/usr/hdp/<version>/oozie". Move/remove "/usr/hdp/current/oozie-client" directory out of "/usr/hdp" and refresh the oozie client from ambari and you should be good.
Created ‎10-06-2018 08:59 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
"/usr/hdp/current/oozie-client" should be a symlink of "/usr/hdp/<version>/oozie". Move/remove "/usr/hdp/current/oozie-client" directory out of "/usr/hdp" and refresh the oozie client from ambari and you should be good.
