Support Questions

Find answers, ask questions, and share your expertise

REST API for starting Spark jobs?

avatar

Does Hortonworks provide a REST API for starting Spark jobs? Microsoft and Cloudera are both looking at Livy. Wondering if Hortonworks has similar plans to provide a tool to make it easy to start Spark jobs via REST.

1 ACCEPTED SOLUTION

avatar

LIVY is started on port 8998, just validated it on my HDP2.5 sandbox.

[root@sandbox ~]# curl localhost:8998/sessions
{"from":0,"total":0,"sessions":[]}

View solution in original post

8 REPLIES 8

avatar

Livy is included in HDP 2.5, although we don't expose it directly (yet). It is used by Zeppelin to support multitenancy and identity propagation.

avatar

Do you know if HortonWorks plans to expose Livy at some point? Any case, I am thinking about installing Livy on my HortonWorks environment in order to take advantage of it. Do you see any possible problems with doing so since HDP2.5 already has Livy embedded somewhere?

avatar

@Carlos Barichello HDP 2.5 has livy embedded , from a support perspective we dont officially support hitting LIVY directly, only through Zeppelin. That being said zeppelin uses the same rest api's to interact with livy.

avatar
@Carlos Barichello

Livy isn't hidden. If you've started Livy, you can use its REST API to launch Spark jobs from Zeppelin or from elsewhere.

avatar

LIVY is started on port 8998, just validated it on my HDP2.5 sandbox.

[root@sandbox ~]# curl localhost:8998/sessions
{"from":0,"total":0,"sessions":[]}

avatar
New Contributor

I tried the same command and I got "Connection refused". Do you have any idea why? Thanks

avatar

Perhaps Livy is not started? Try running this in the command window:

/usr/hdp/2.5.0.0-817/livy/bin/livy-server

avatar

Thanks everybody for your postings. I downloaded the HDP 2.5 Sandbox, tested out Livy, and things worked fine for the most part. It would be great if Hortonworks could make Livy official for the 2.5 release along with some great documentation.