question Re: REST API for starting Spark jobs? in Support Questions

REST API for starting Spark jobs?

carlos_barichel — Wed, 14 Sep 2016 22:52:36 GMT

Does Hortonworks provide a REST API for starting Spark jobs? Microsoft and Cloudera are both looking at Livy. Wondering if Hortonworks has similar plans to provide a tool to make it easy to start Spark jobs via REST.

Re: REST API for starting Spark jobs?

Former Member — Wed, 14 Sep 2016 22:57:26 GMT

Livy is included in HDP 2.5, although we don't expose it directly (yet). It is used by Zeppelin to support multitenancy and identity propagation.

Re: REST API for starting Spark jobs?

carlos_barichel — Wed, 14 Sep 2016 23:02:53 GMT

Do you know if HortonWorks plans to expose Livy at some point? Any case, I am thinking about installing Livy on my HortonWorks environment in order to take advantage of it. Do you see any possible problems with doing so since HDP2.5 already has Livy embedded somewhere?

Re: REST API for starting Spark jobs?

azeltov — Thu, 15 Sep 2016 00:35:53 GMT

@Carlos Barichello HDP 2.5 has livy embedded , from a support perspective we dont officially support hitting LIVY directly, only through Zeppelin. That being said zeppelin uses the same rest api's to interact with livy.

Re: REST API for starting Spark jobs?

rgelhausen — Thu, 15 Sep 2016 01:29:43 GMT

@Carlos Barichello

Livy isn't hidden. If you've started Livy, you can use its REST API to launch Spark jobs from Zeppelin or from elsewhere.

Re: REST API for starting Spark jobs?

azeltov — Thu, 15 Sep 2016 01:38:45 GMT

LIVY is started on port 8998, just validated it on my HDP2.5 sandbox.

[root@sandbox ~]# curl localhost:8998/sessions
{"from":0,"total":0,"sessions":[]}

Re: REST API for starting Spark jobs?

carlos_barichel — Fri, 16 Sep 2016 04:28:19 GMT

Thanks everybody for your postings. I downloaded the HDP 2.5 Sandbox, tested out Livy, and things worked fine for the most part. It would be great if Hortonworks could make Livy official for the 2.5 release along with some great documentation.

Re: REST API for starting Spark jobs?

yramos — Wed, 12 Oct 2016 00:09:58 GMT

I tried the same command and I got "Connection refused". Do you have any idea why? Thanks

Re: REST API for starting Spark jobs?

carlos_barichel — Wed, 12 Oct 2016 01:45:23 GMT

Perhaps Livy is not started? Try running this in the command window:

/usr/hdp/2.5.0.0-817/livy/bin/livy-server