question Re: spark-shell vs spark-submit command in Archives of Support Questions (Read Only)

spark-shell vs spark-submit command

avi166 — Fri, 15 Jun 2018 22:38:54 GMT

We are running spark jobs using spark-shell command :

e.g: spark-shell --conf config_file_details --driver-memory 4G --executor-memory 4G -i spark_job.scala

we are not using spark-submit command to submit job but instead of it we are using spark-shell command

Could you please advice which will be fast in terms of performance.

Thanks,

Re: spark-shell vs spark-submit command

falbani — Fri, 15 Jun 2018 23:16:51 GMT

@Avinash A Spark shell is only intended to be use for testing and perhaps development of small applications - is only an interactive shell and should not be use to run production spark applications. For production application deployment you should use spark-submit. The last one will also allow you to run applications in yarn-cluster mode.

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: spark-shell vs spark-submit command

falbani — Mon, 18 Jun 2018 20:33:38 GMT

@Avinash A if the above answer helped you please take a moment to login and click the "accept" link on the answer.

Re: spark-shell vs spark-submit command

avi166 — Mon, 18 Jun 2018 21:02:27 GMT

Thanks a lot! Info was very useful.