Support Questions
Find answers, ask questions, and share your expertise

Livy Spark Rest Jar submission interactive session

Livy Spark Rest Jar submission interactive session

New Contributor

Hi All,

 

I can see livy documentation (https://livy.incubator.apache.org/docs/latest/rest-api.html) has an option to upload a jar file while creating an interactive session.

 

What I am trying - 

curl --silent --negotiate -u:<user> http://<host>:8996/sessions -X POST -H 'Content-Type: application/json' -d
'{
"kind":"spark",
"proxyUser":"<user>",
"jars": "<jar_file_path_in_HDFS>", //Option I tried: Jar from HDFS relative path, absolute path, local path mentioned in livy conf file (livy.file.local-dir-whitelis)
"name":"TestSparkScalaSession"
}'
{"id":0,"name":"TestSparkScalaSession","appId":null,"owner":null,"proxyUser":"<user>","state":"starting","kind":"spark",
"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: ","\nYARN Diagnostics: "],"startTime":-1,"endTime":-1,"elapsedTime":0}

 

Next I am trying to execute some scala code 

curl --silent --negotiate -u:<user> http://<host>:8996/sessions/TestSparkScalaSession/statements -X POST -H 'Content-Type: application/json' -d
'{"code":"import com.test.spark.TestReportData; val empData = new TestReportData; empData.run(Array.empty[String])"}'


{"total_statements":1,"statements":[{"id":0,"state":"available","output":{"status":"error","execution_count":0,"ename":"Error","evalue":"<console>:23: error: object test is not a member of package com","traceback":[" import com.test.spark.TestReportData; val empData = new TestReportData;;\n"," ^\n","<console>:23: error: not found: type TestReportData\n"," import com.test.spark.TestReportData; val empData = new TestReportData;;\n",

 

So it can't read or import classes from jar file.  

 

In livy ui I can see  

20/09/14 10:19:16 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://e54fbcdd1f92:4043
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-api-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-api-0.6.0-incubating.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/netty-all-4.0.37.Final.jar at spark://e54fbcdd1f92:38395/jars/netty-all-4.0.37.Final.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-rsc-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-rsc-0.6.0-incubating.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-thriftserver-session-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-thriftserver-session-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/livy-core_2.11-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-core_2.11-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/livy-repl_2.11-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-repl_2.11-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/commons-codec-1.9.jar at spark://e54fbcdd1f92:38395/jars/commons-codec-1.9.jar with timestamp 1600078756322

I don't see uploaded jar file attached to spark context.

Is there anything wrong I am doing. 

In Batch its working fine.

Any help/suggestion will be highly appreciated.

Regards,
Badal