Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

REST API for starting Spark jobs?

Does Hortonworks provide a REST API for starting Spark jobs? Microsoft and Cloudera are both looking at Livy. Wondering if Hortonworks has similar plans to provide a tool to make it easy to start Spark jobs via REST.

1 ACCEPTED SOLUTION

LIVY is started on port 8998, just validated it on my HDP2.5 sandbox.

[root@sandbox ~]# curl localhost:8998/sessions
{"from":0,"total":0,"sessions":[]}

View solution in original post

8 REPLIES 8

Livy is included in HDP 2.5, although we don't expose it directly (yet). It is used by Zeppelin to support multitenancy and identity propagation.

Do you know if HortonWorks plans to expose Livy at some point? Any case, I am thinking about installing Livy on my HortonWorks environment in order to take advantage of it. Do you see any possible problems with doing so since HDP2.5 already has Livy embedded somewhere?

@Carlos Barichello HDP 2.5 has livy embedded , from a support perspective we dont officially support hitting LIVY directly, only through Zeppelin. That being said zeppelin uses the same rest api's to interact with livy.

@Carlos Barichello

Livy isn't hidden. If you've started Livy, you can use its REST API to launch Spark jobs from Zeppelin or from elsewhere.

LIVY is started on port 8998, just validated it on my HDP2.5 sandbox.

[root@sandbox ~]# curl localhost:8998/sessions
{"from":0,"total":0,"sessions":[]}

New Contributor

I tried the same command and I got "Connection refused". Do you have any idea why? Thanks

Perhaps Livy is not started? Try running this in the command window:

/usr/hdp/2.5.0.0-817/livy/bin/livy-server

Thanks everybody for your postings. I downloaded the HDP 2.5 Sandbox, tested out Livy, and things worked fine for the most part. It would be great if Hortonworks could make Livy official for the 2.5 release along with some great documentation.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.