Member since
09-15-2020
1
Post
0
Kudos Received
0
Solutions
09-15-2020
10:55 PM
Hi All, I can see livy documentation (https://livy.incubator.apache.org/docs/latest/rest-api.html) has an option to upload a jar file while creating an interactive session. What I am trying - curl --silent --negotiate -u:<user> http://<host>:8996/sessions -X POST -H 'Content-Type: application/json' -d '{ "kind":"spark", "proxyUser":"<user>", "jars": "<jar_file_path_in_HDFS>", //Option I tried: Jar from HDFS relative path, absolute path, local path mentioned in livy conf file (livy.file.local-dir-whitelis) "name":"TestSparkScalaSession" }' {"id":0,"name":"TestSparkScalaSession","appId":null,"owner":null,"proxyUser":"<user>","state":"starting","kind":"spark", "appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: ","\nYARN Diagnostics: "],"startTime":-1,"endTime":-1,"elapsedTime":0} Next I am trying to execute some scala code curl --silent --negotiate -u:<user> http://<host>:8996/sessions/TestSparkScalaSession/statements -X POST -H 'Content-Type: application/json' -d '{"code":"import com.test.spark.TestReportData; val empData = new TestReportData; empData.run(Array.empty[String])"}' {"total_statements":1,"statements":[{"id":0,"state":"available","output":{"status":"error","execution_count":0,"ename":"Error","evalue":"<console>:23: error: object test is not a member of package com","traceback":[" import com.test.spark.TestReportData; val empData = new TestReportData;;\n"," ^\n","<console>:23: error: not found: type TestReportData\n"," import com.test.spark.TestReportData; val empData = new TestReportData;;\n", So it can't read or import classes from jar file. In livy ui I can see 20/09/14 10:19:16 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://e54fbcdd1f92:4043 20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-api-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-api-0.6.0-incubating.jar with timestamp 1600078756321 20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/netty-all-4.0.37.Final.jar at spark://e54fbcdd1f92:38395/jars/netty-all-4.0.37.Final.jar with timestamp 1600078756321 20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-rsc-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-rsc-0.6.0-incubating.jar with timestamp 1600078756321 20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-thriftserver-session-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-thriftserver-session-0.6.0-incubating.jar with timestamp 1600078756322 20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/livy-core_2.11-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-core_2.11-0.6.0-incubating.jar with timestamp 1600078756322 20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/livy-repl_2.11-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-repl_2.11-0.6.0-incubating.jar with timestamp 1600078756322 20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/commons-codec-1.9.jar at spark://e54fbcdd1f92:38395/jars/commons-codec-1.9.jar with timestamp 1600078756322 I don't see uploaded jar file attached to spark context. Is there anything wrong I am doing. In Batch its working fine. Any help/suggestion will be highly appreciated. Regards, Badal
... View more
Labels: