How to post a Spark Job as JAR via Livy interactive REST interface

[#1]

I am new to Livy. I am able to run scala code through Livy api POST statement as described here -

https://livy.incubator.apache.org/examples/

How can I submit a Jar file using API?

[#2 · Accepted solution]

@Mukesh Chouhan

Try adding jars using the jars option while posting to session like described in the livy rest documentation:

https://livy.incubator.apache.org/docs/latest/rest-api.html

curl -X POST -d '{"conf": {"jars": "hdfs://localhost:8020/tmp/package.jar"}}' -H "Content-Type: application/json" localhost:8998/sessions

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

[#3]

@Mukesh Chouhan the above example is pointing to hdfs location for the jars. Is it possible for you to do the same and upload your jars to an hdfs location. Then point to the hdfs location as I'm doing above?

HTH

[#4]

Thanks @Felix Albani I will try that.

Is it possible to upload jar file which is present locally on my server from where I executing curl?

[#5]

@Mukesh Chouhan AFAIK you can't submit your jars along with the code using the livy api. You need to place them in HDFS or Livy local file system in advanced.

Please if the above answers have helped remember to login and mark as Accepted.

[#6]

@Felix Albani I have tried below curl commands-

curl -X POST -d
'{"className": "com.company.scala.ScyllaSpanByTest" ,
"file":
"hdfs://dhdfsserver:8020/user/spark/jars/Test-assembly-0.1.jar"}'
-H "Content-Type: application/json" -H "X-Requested-By:
user" http://livyserver:8999/batches

curl -X POST -d
'{"conf": {"kind": "spark"
,"className": "com.company.scala.ScyllaSpanByTest" ,
"jars":
"hdfs://hdfsserver:8020/user/spark/jars/Test-assembly-0.1.jar"
}}' -H "Content-Type: application/json" -H "X-Requested-By:
user" http://livyserver:8999/sessions

[#7]

second curl command which is for sessions may not work as there is no property called className in sessions REST API

[#8]

hi,

Is this solution working for you?

I'm not able to upload the jar to the session in any way.

Uploading the same jar to a batch is working though.

Tried using :

hdfs://localhost:8020..,

hdfs://..

hdfs://...

file:///

s3://

Nothing seems to be working.

Maybe you have any reason why?

[#9]

How to import scala object and call the main funtion in interactive using Livy REST API

[#10]

Thanks for your quick reply @Felix Albani

Suppose me Livy Server IP is on X.X.X.X (port 8999) and I am executing CURL from server with Y.Y.Y.Y

My jar file is present on server Y.Y.Y.Y at location /home/app/work

Does below command looks good to you?

curl -X POST -d '{"conf": {"kind": "spark" , "jars": "hdfs://Y.Y.Y.Y:8020/home/app/work/scalajar/helloworld.jar"}}' -H "Content-Type: application/json" -H "X-Requested-By: user" http://X.X.X.X:8999/sessions

Also can I specify "kind": "spark" as above in my curl command?