Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to launch spark application without using spark-submit command

how to launch spark application without using spark-submit command

Hi

 

Can we launch Spark Application without using spark-submit command . as there any other method means using rest api or some other framework . so we can launch spark application from outside of cluster.

 

Regards

Prateek

4 REPLIES 4

Re: how to launch spark application without using spark-submit command

Explorer

Re: how to launch spark application without using spark-submit command

is cloudera 5.5.1 hue have support of Livy spark rest job server ?

Re: how to launch spark application without using spark-submit command

Explorer

According to the following page, hue supported spark notebook(frontend of livy server) since 3.8.

http://gethue.com/new-notebook-application-for-spark-sql/

 

CDH hue might have become 3.9 since 5.5.0

https://archive.cloudera.com/cdh5/cdh/5/hue-3.9.0-cdh5.5.0.releasenotes.html

 

so, CDH 5.5.1 seems to have livy server.

 

However, even if cdh 5.6.0, spark notebook seems to be still experimental.

If you use cdh hue, livy server can be booted in foreground process with the following command.

 

# sudo -u hue /usr/lib/hue/build/env/bin/hue livy_server

 

 

Re: how to launch spark application without using spark-submit command

Rising Star

Another option would be to use Oozie which now has a Spark action.

 

Regards,

Martin

Don't have an account?
Coming from Hortonworks? Activate your account here