Reply
New Contributor
Posts: 2
Registered: ‎10-23-2018

spark-submit with sbt generated file failed

[ Edited ]

Hi,

 

I use cloudera with spark v 1.6.0 and scala v 10.5, installed.

I run sbt package command and the jar simple-programme_2.10-1.0.jar was generated.

But when I run spark-submit command (see below), it failed. Have you any solutions please?

 

I used the file SimpleProg which contain the following text:

name := "Simple Programme"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.6.0"

 

------------------------------------------------------------------

Here is the spark submit command:

 

[cloudera@quickstart ~]$ cd
[cloudera@quickstart ~]$ cd tp2_num
[cloudera@quickstart tp2_num]$ spark-submit --class "SimpleProg" --master local target/scala-2.10/simple-programme_2.10-1.0.jar
java.lang.ClassNotFoundException: SimpleProg
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:177)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:688)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[cloudera@quickstart tp2_num]$

Highlighted
New Contributor
Posts: 2
Registered: ‎10-23-2018

Re: spark-submit with sbt generated file failed

I attach the program content that is in /home/cloudera/tp2-num/src/main/scala/SimpleProg.scala

Announcements