Support Questions
Find answers, ask questions, and share your expertise

REST API for starting Spark jobs?

Does Hortonworks provide a REST API for starting Spark jobs? Microsoft and Cloudera are both looking at Livy. Wondering if Hortonworks has similar plans to provide a tool to make it easy to start Spark jobs via REST.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: REST API for starting Spark jobs?

LIVY is started on port 8998, just validated it on my HDP2.5 sandbox.

[root@sandbox ~]# curl localhost:8998/sessions
{"from":0,"total":0,"sessions":[]}

View solution in original post

8 REPLIES 8

Re: REST API for starting Spark jobs?

Livy is included in HDP 2.5, although we don't expose it directly (yet). It is used by Zeppelin to support multitenancy and identity propagation.

Re: REST API for starting Spark jobs?

Do you know if HortonWorks plans to expose Livy at some point? Any case, I am thinking about installing Livy on my HortonWorks environment in order to take advantage of it. Do you see any possible problems with doing so since HDP2.5 already has Livy embedded somewhere?

Re: REST API for starting Spark jobs?

@Carlos Barichello HDP 2.5 has livy embedded , from a support perspective we dont officially support hitting LIVY directly, only through Zeppelin. That being said zeppelin uses the same rest api's to interact with livy.

Re: REST API for starting Spark jobs?

@Carlos Barichello

Livy isn't hidden. If you've started Livy, you can use its REST API to launch Spark jobs from Zeppelin or from elsewhere.

Re: REST API for starting Spark jobs?

LIVY is started on port 8998, just validated it on my HDP2.5 sandbox.

[root@sandbox ~]# curl localhost:8998/sessions
{"from":0,"total":0,"sessions":[]}

View solution in original post

Re: REST API for starting Spark jobs?

New Contributor

I tried the same command and I got "Connection refused". Do you have any idea why? Thanks

Re: REST API for starting Spark jobs?

Perhaps Livy is not started? Try running this in the command window:

/usr/hdp/2.5.0.0-817/livy/bin/livy-server

Re: REST API for starting Spark jobs?

Thanks everybody for your postings. I downloaded the HDP 2.5 Sandbox, tested out Livy, and things worked fine for the most part. It would be great if Hortonworks could make Livy official for the 2.5 release along with some great documentation.