Hi All,
I can see livy documentation (https://livy.incubator.apache.org/docs/latest/rest-api.html) has an option to upload a jar file while creating an interactive session.
What I am trying -
curl --silent --negotiate -u:<user> http://<host>:8996/sessions -X POST -H 'Content-Type: application/json' -d
'{
"kind":"spark",
"proxyUser":"<user>",
"jars": "<jar_file_path_in_HDFS>", //Option I tried: Jar from HDFS relative path, absolute path, local path mentioned in livy conf file (livy.file.local-dir-whitelis)
"name":"TestSparkScalaSession"
}'
{"id":0,"name":"TestSparkScalaSession","appId":null,"owner":null,"proxyUser":"<user>","state":"starting","kind":"spark",
"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: ","\nYARN Diagnostics: "],"startTime":-1,"endTime":-1,"elapsedTime":0}
Next I am trying to execute some scala code
curl --silent --negotiate -u:<user> http://<host>:8996/sessions/TestSparkScalaSession/statements -X POST -H 'Content-Type: application/json' -d
'{"code":"import com.test.spark.TestReportData; val empData = new TestReportData; empData.run(Array.empty[String])"}'
{"total_statements":1,"statements":[{"id":0,"state":"available","output":{"status":"error","execution_count":0,"ename":"Error","evalue":"<console>:23: error: object test is not a member of package com","traceback":[" import com.test.spark.TestReportData; val empData = new TestReportData;;\n"," ^\n","<console>:23: error: not found: type TestReportData\n"," import com.test.spark.TestReportData; val empData = new TestReportData;;\n",
So it can't read or import classes from jar file.
In livy ui I can see
20/09/14 10:19:16 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://e54fbcdd1f92:4043
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-api-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-api-0.6.0-incubating.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/netty-all-4.0.37.Final.jar at spark://e54fbcdd1f92:38395/jars/netty-all-4.0.37.Final.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-rsc-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-rsc-0.6.0-incubating.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-thriftserver-session-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-thriftserver-session-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/livy-core_2.11-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-core_2.11-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/livy-repl_2.11-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-repl_2.11-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/commons-codec-1.9.jar at spark://e54fbcdd1f92:38395/jars/commons-codec-1.9.jar with timestamp 1600078756322
I don't see uploaded jar file attached to spark context.
Is there anything wrong I am doing.
In Batch its working fine.
Any help/suggestion will be highly appreciated.
Regards,
Badal
