Support Questions
Find answers, ask questions, and share your expertise

how to Spark submit command with Scala

how to Spark submit command with Scala

Explorer

I an new bee in spark. I have create a Scala script.

I can run this script by "spark-shell -p my_script.scala "

how to use spark-submit to use scala script ?

2 REPLIES 2

Re: how to Spark submit command with Scala

spark-shell is scala repl which you can use to run and test scala expression and scala script. spark-submit is an utility provided by spark to submit spark job to the cluster. further on this you can read this http://spark.apache.org/docs/latest/submitting-applications.html

Re: how to Spark submit command with Scala

You need to compile a spark application. There are different ways out there to do it. SBT is perhaps the most common one. With IntelliJ ahead of Scala-IDE if you want an IDE. ( You don't need one you can use vi and sbt/maven as well if you want to suffer)

http://spark.apache.org/docs/latest/building-spark.html

Look at the spark examples to get an idea on how to structure an application. ( You need a main class for example.)

https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples