Reply
Contributor
Posts: 43
Registered: ‎03-04-2015

how to launch spark application without using spark-submit command

Hi

 

Can we launch Spark Application without using spark-submit command . as there any other method means using rest api or some other framework . so we can launch spark application from outside of cluster.

 

Regards

Prateek

Explorer
Posts: 11
Registered: ‎11-25-2015

Re: how to launch spark application without using spark-submit command

Contributor
Posts: 43
Registered: ‎03-04-2015

Re: how to launch spark application without using spark-submit command

is cloudera 5.5.1 hue have support of Livy spark rest job server ?
Explorer
Posts: 11
Registered: ‎11-25-2015

Re: how to launch spark application without using spark-submit command

[ Edited ]

According to the following page, hue supported spark notebook(frontend of livy server) since 3.8.

http://gethue.com/new-notebook-application-for-spark-sql/

 

CDH hue might have become 3.9 since 5.5.0

https://archive.cloudera.com/cdh5/cdh/5/hue-3.9.0-cdh5.5.0.releasenotes.html

 

so, CDH 5.5.1 seems to have livy server.

 

However, even if cdh 5.6.0, spark notebook seems to be still experimental.

If you use cdh hue, livy server can be booted in foreground process with the following command.

 

# sudo -u hue /usr/lib/hue/build/env/bin/hue livy_server

 

 

Contributor
Posts: 58
Registered: ‎10-19-2014

Re: how to launch spark application without using spark-submit command

Another option would be to use Oozie which now has a Spark action.

 

Regards,

Martin