Support Questions
Find answers, ask questions, and share your expertise

Spark Livy Server script submission and result set

Spark Livy Server script submission and result set

When submitting a script to Livy (vs using interactive Scala or Python), is there a way to make 1 call and get back the results in that 1 call?

Typically, script submission such as in submitting a PySpark job is for batch processing and the immediate JSON result set will give information about the session ID. For example, when I submit a PySpark script to Live I get back the following:

13495-yql1p.png

I then take the Session ID from the above output and make a second call to get back the log file which contains the output data:

curl my_Livy_Server:8998/batches/35/log | python -m json.tool

Is there a way to only make 1 call and get back the result set in that 1 call?

1 REPLY 1

Re: Spark Livy Server script submission and result set

Expert Contributor

There's no such api for this. 2 solutions for you

1. Wrap the submission and pulling api by yourself

2. Use zeppelin rest api, zeppelin has already integrated livy interpreter. And zeppelin provides synchronously for running a paragraph. So that you can just call one zeppelin rest api to get the result.

https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/rest-api/rest-notebook.html#run-a-paragraph-synchron...

https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/interpreter/livy.html