Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to Spark submit command with Scala

Highlighted

how to Spark submit command with Scala

Explorer

I an new bee in spark. I have create a Scala script.

I can run this script by "spark-shell -p my_script.scala "

how to use spark-submit to use scala script ?

2 REPLIES 2
Highlighted

Re: how to Spark submit command with Scala

spark-shell is scala repl which you can use to run and test scala expression and scala script. spark-submit is an utility provided by spark to submit spark job to the cluster. further on this you can read this http://spark.apache.org/docs/latest/submitting-applications.html

Highlighted

Re: how to Spark submit command with Scala

You need to compile a spark application. There are different ways out there to do it. SBT is perhaps the most common one. With IntelliJ ahead of Scala-IDE if you want an IDE. ( You don't need one you can use vi and sbt/maven as well if you want to suffer)

http://spark.apache.org/docs/latest/building-spark.html

Look at the spark examples to get an idea on how to structure an application. ( You need a main class for example.)

https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples

Don't have an account?
Coming from Hortonworks? Activate your account here