question Re: How to post a Spark Job as JAR via Livy interactive REST interface in Support Questions

How to post a Spark Job as JAR via Livy interactive REST interface

mukesh_chouhan — Sun, 15 Jul 2018 20:00:39 GMT

I am new to Livy. I am able to run scala code through Livy api POST statement as described here -

https://livy.incubator.apache.org/examples/

How can I submit a Jar file using API?

Re: How to post a Spark Job as JAR via Livy interactive REST interface

falbani — Sun, 15 Jul 2018 20:27:17 GMT

@Mukesh Chouhan

Try adding jars using the jars option while posting to session like described in the livy rest documentation:

https://livy.incubator.apache.org/docs/latest/rest-api.html

curl -X POST -d '{"conf": {"jars": "hdfs://localhost:8020/tmp/package.jar"}}' -H "Content-Type: application/json" localhost:8998/sessions

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: How to post a Spark Job as JAR via Livy interactive REST interface

mukesh_chouhan — Sun, 15 Jul 2018 22:35:37 GMT

Thanks for your quick reply @Felix Albani

Suppose me Livy Server IP is on X.X.X.X (port 8999) and I am executing CURL from server with Y.Y.Y.Y

My jar file is present on server Y.Y.Y.Y at location /home/app/work

Does below command looks good to you?

curl -X POST -d '{"conf": {"kind": "spark" , "jars": "hdfs://Y.Y.Y.Y:8020/home/app/work/scalajar/helloworld.jar"}}' -H "Content-Type: application/json" -H "X-Requested-By: user" http://X.X.X.X:8999/sessions

Also can I specify "kind": "spark" as above in my curl command?

Re: How to post a Spark Job as JAR via Livy interactive REST interface

falbani — Sun, 15 Jul 2018 23:16:46 GMT

@Mukesh Chouhan the above example is pointing to hdfs location for the jars. Is it possible for you to do the same and upload your jars to an hdfs location. Then point to the hdfs location as I'm doing above?

HTH

Re: How to post a Spark Job as JAR via Livy interactive REST interface

mukesh_chouhan — Mon, 16 Jul 2018 11:51:52 GMT

Thanks @Felix Albani I will try that.

Is it possible to upload jar file which is present locally on my server from where I executing curl?

Re: How to post a Spark Job as JAR via Livy interactive REST interface

sandyy006 — Mon, 16 Jul 2018 15:17:45 GMT

@Mukesh Chouhan

In order to use the jars present in local filesystem. Please follow below.

1. Place the jars in a directory on livy node and add the directory to `livy.file.local-dir-whitelist`.This configuration should be set in livy.conf.

2. Add all the required jars to "jars" field in the curl command, note it should be added in URI format with "file" scheme, like "file://<livy.file.local-dir-whitelist>/xxx.jar".

Re: How to post a Spark Job as JAR via Livy interactive REST interface

falbani — Mon, 16 Jul 2018 19:50:30 GMT

@Mukesh Chouhan AFAIK you can't submit your jars along with the code using the livy api. You need to place them in HDFS or Livy local file system in advanced.

Please if the above answers have helped remember to login and mark as Accepted.

Re: How to post a Spark Job as JAR via Livy interactive REST interface

mukesh_chouhan — Wed, 18 Jul 2018 20:41:31 GMT

@Felix Albani I have tried below curl commands-

curl -X POST -d
'{"className": "com.company.scala.ScyllaSpanByTest" ,
"file":
"hdfs://dhdfsserver:8020/user/spark/jars/Test-assembly-0.1.jar"}'
-H "Content-Type: application/json" -H "X-Requested-By:
user" http://livyserver:8999/batches

curl -X POST -d
'{"conf": {"kind": "spark"
,"className": "com.company.scala.ScyllaSpanByTest" ,
"jars":
"hdfs://hdfsserver:8020/user/spark/jars/Test-assembly-0.1.jar"
}}' -H "Content-Type: application/json" -H "X-Requested-By:
user" http://livyserver:8999/sessions

Re: How to post a Spark Job as JAR via Livy interactive REST interface

eliadv — Thu, 16 Jul 2020 07:49:03 GMT

hi,

Is this solution working for you?

I'm not able to upload the jar to the session in any way.

Uploading the same jar to a batch is working though.

Tried using :

hdfs://localhost:8020..,

hdfs://..

hdfs://...

file:///

s3://

Nothing seems to be working.

Maybe you have any reason why?

Re: How to post a Spark Job as JAR via Livy interactive REST interface

mana — Sun, 30 May 2021 03:50:09 GMT

How to import scala object and call the main funtion in interactive using Livy REST API

Re: How to post a Spark Job as JAR via Livy interactive REST interface

mana — Sun, 30 May 2021 03:53:18 GMT

second curl command which is for sessions may not work as there is no property called className in sessions REST API