REST API for starting Spark jobs?

[#1]

Does Hortonworks provide a REST API for starting Spark jobs? Microsoft and Cloudera are both looking at Livy. Wondering if Hortonworks has similar plans to provide a tool to make it easy to start Spark jobs via REST.

[#2]

Livy is included in HDP 2.5, although we don't expose it directly (yet). It is used by Zeppelin to support multitenancy and identity propagation.

[#3]

Do you know if HortonWorks plans to expose Livy at some point? Any case, I am thinking about installing Livy on my HortonWorks environment in order to take advantage of it. Do you see any possible problems with doing so since HDP2.5 already has Livy embedded somewhere?

[#4]

@Carlos Barichello HDP 2.5 has livy embedded , from a support perspective we dont officially support hitting LIVY directly, only through Zeppelin. That being said zeppelin uses the same rest api's to interact with livy.

[#5]

@Carlos Barichello

Livy isn't hidden. If you've started Livy, you can use its REST API to launch Spark jobs from Zeppelin or from elsewhere.

[#6 · Accepted solution]

LIVY is started on port 8998, just validated it on my HDP2.5 sandbox.

[root@sandbox ~]# curl localhost:8998/sessions
{"from":0,"total":0,"sessions":[]}

[#7]

I tried the same command and I got "Connection refused". Do you have any idea why? Thanks

[#8]

Perhaps Livy is not started? Try running this in the command window:

/usr/hdp/2.5.0.0-817/livy/bin/livy-server

[#9]

Thanks everybody for your postings. I downloaded the HDP 2.5 Sandbox, tested out Livy, and things worked fine for the most part. It would be great if Hortonworks could make Livy official for the 2.5 release along with some great documentation.