Support Questions
Find answers, ask questions, and share your expertise

How to post a Spark Job as JAR via Livy interactive REST interface

New Contributor

I am new to Livy. I am able to run scala code through Livy api POST statement as described here -

https://livy.incubator.apache.org/examples/

How can I submit a Jar file using API?

1 ACCEPTED SOLUTION

Accepted Solutions

@Mukesh Chouhan

Try adding jars using the jars option while posting to session like described in the livy rest documentation:

https://livy.incubator.apache.org/docs/latest/rest-api.html

curl -X POST -d '{"conf": {"jars": "hdfs://localhost:8020/tmp/package.jar"}}' -H "Content-Type: application/json" localhost:8998/sessions

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

View solution in original post

10 REPLIES 10

@Mukesh Chouhan

Try adding jars using the jars option while posting to session like described in the livy rest documentation:

https://livy.incubator.apache.org/docs/latest/rest-api.html

curl -X POST -d '{"conf": {"jars": "hdfs://localhost:8020/tmp/package.jar"}}' -H "Content-Type: application/json" localhost:8998/sessions

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

View solution in original post

@Mukesh Chouhan the above example is pointing to hdfs location for the jars. Is it possible for you to do the same and upload your jars to an hdfs location. Then point to the hdfs location as I'm doing above?

HTH

New Contributor

Thanks @Felix Albani I will try that.

Is it possible to upload jar file which is present locally on my server from where I executing curl?

@Mukesh Chouhan AFAIK you can't submit your jars along with the code using the livy api. You need to place them in HDFS or Livy local file system in advanced.

Please if the above answers have helped remember to login and mark as Accepted.

New Contributor

@Felix Albani I have tried below curl commands-

curl -X POST -d
'{"className": "com.company.scala.ScyllaSpanByTest" ,
"file":
"hdfs://dhdfsserver:8020/user/spark/jars/Test-assembly-0.1.jar"}'
-H "Content-Type: application/json" -H "X-Requested-By:
user" http://livyserver:8999/batches

curl -X POST -d
'{"conf": {"kind": "spark"
,"className": "com.company.scala.ScyllaSpanByTest" ,
"jars":
"hdfs://hdfsserver:8020/user/spark/jars/Test-assembly-0.1.jar"
}}' -H "Content-Type: application/json" -H "X-Requested-By:
user" http://livyserver:8999/sessions

New Contributor

second curl command which is for sessions may not work as there is no property called className in sessions REST API

New Contributor

hi,

 

Is this solution working for you?

I'm not able to upload the jar to the session in any way.

Uploading the same jar to a batch is working though.

Tried using :

hdfs://localhost:8020..,

hdfs://..

hdfs://...

file:///

s3://

Nothing seems to be working.

Maybe you have any reason why?

New Contributor

How to import scala object and call the main funtion in interactive using Livy REST API

New Contributor

Thanks for your quick reply @Felix Albani

Suppose me Livy Server IP is on X.X.X.X (port 8999) and I am executing CURL from server with Y.Y.Y.Y

My jar file is present on server Y.Y.Y.Y at location /home/app/work

Does below command looks good to you?

curl -X POST -d '{"conf": {"kind": "spark" , "jars": "hdfs://Y.Y.Y.Y:8020/home/app/work/scalajar/helloworld.jar"}}' -H "Content-Type: application/json" -H "X-Requested-By: user" http://X.X.X.X:8999/sessions

Also can I specify "kind": "spark" as above in my curl command?