question Re: Jars files using Yarn in Spark in Support Questions

Jars files using Yarn in Spark

harshilgala37 — Thu, 29 Jun 2017 11:42:29 GMT

Hi,

I have a .jar file in one of the location (local or cluster) and I want to run that file using spark-shell in Yarn mode.. Can any one help me with this??

Thanks

Re: Jars files using Yarn in Spark

MindGlass — Thu, 29 Jun 2017 15:21:49 GMT

Try like this command.

spark-shell --jars /app/spark/a.jar,/app/spark/b.jar

Re: Jars files using Yarn in Spark

harshilgala37 — Thu, 29 Jun 2017 16:37:19 GMT

Thanks Peter Kim,

Correct me if I my understanding is wrong,

spark-shell --jars "location" of jar files --master "yarn"

is this command correct? Do I need to write anything else along with --master "yarn" this for Yarn

Re: Jars files using Yarn in Spark

sandyy006 — Fri, 30 Jun 2017 02:05:06 GMT

@Harshil Gala

spark-shell --jars <path-to-jar> --master yarn --deploy-mode client/cluster (you can choose client or accordingly)

Re: Jars files using Yarn in Spark

MindGlass — Fri, 30 Jun 2017 13:41:22 GMT

If you're using hadoop cluster with ambari of hortonworks, then you don't have to use that --master yarn parameter. Cause' spark service mode of HDP cluster is installed to yarn mode basically.