Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to Spark submit command with Scala

how to Spark submit command with Scala

New Contributor

I an new bee in spark. I have create a Scala script.

I can run this script by "spark-shell -p my_script.scala "

how to use spark-submit to use scala script ?

2 REPLIES 2

Re: how to Spark submit command with Scala

spark-shell is scala repl which you can use to run and test scala expression and scala script. spark-submit is an utility provided by spark to submit spark job to the cluster. further on this you can read this http://spark.apache.org/docs/latest/submitting-applications.html

Re: how to Spark submit command with Scala

You need to compile a spark application. There are different ways out there to do it. SBT is perhaps the most common one. With IntelliJ ahead of Scala-IDE if you want an IDE. ( You don't need one you can use vi and sbt/maven as well if you want to suffer)

http://spark.apache.org/docs/latest/building-spark.html

Look at the spark examples to get an idea on how to structure an application. ( You need a main class for example.)

https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples