Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Livy Spark Rest Jar submission interactive session

Livy Spark Rest Jar submission interactive session

New Contributor

Hi All,

 

I can see livy documentation (https://livy.incubator.apache.org/docs/latest/rest-api.html) has an option to upload a jar file while creating an interactive session.

 

What I am trying - 

curl --silent --negotiate -u:<user> http://<host>:8996/sessions -X POST -H 'Content-Type: application/json' -d
'{
"kind":"spark",
"proxyUser":"<user>",
"jars": "<jar_file_path_in_HDFS>", //Option I tried: Jar from HDFS relative path, absolute path, local path mentioned in livy conf file (livy.file.local-dir-whitelis)
"name":"TestSparkScalaSession"
}'
{"id":0,"name":"TestSparkScalaSession","appId":null,"owner":null,"proxyUser":"<user>","state":"starting","kind":"spark",
"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: ","\nYARN Diagnostics: "],"startTime":-1,"endTime":-1,"elapsedTime":0}

 

Next I am trying to execute some scala code 

curl --silent --negotiate -u:<user> http://<host>:8996/sessions/TestSparkScalaSession/statements -X POST -H 'Content-Type: application/json' -d
'{"code":"import com.test.spark.TestReportData; val empData = new TestReportData; empData.run(Array.empty[String])"}'


{"total_statements":1,"statements":[{"id":0,"state":"available","output":{"status":"error","execution_count":0,"ename":"Error","evalue":"<console>:23: error: object test is not a member of package com","traceback":[" import com.test.spark.TestReportData; val empData = new TestReportData;;\n"," ^\n","<console>:23: error: not found: type TestReportData\n"," import com.test.spark.TestReportData; val empData = new TestReportData;;\n",

 

So it can't read or import classes from jar file.  

 

In livy ui I can see  

20/09/14 10:19:16 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://e54fbcdd1f92:4043
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-api-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-api-0.6.0-incubating.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/netty-all-4.0.37.Final.jar at spark://e54fbcdd1f92:38395/jars/netty-all-4.0.37.Final.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-rsc-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-rsc-0.6.0-incubating.jar with timestamp 1600078756321
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/rsc-jars/livy-thriftserver-session-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-thriftserver-session-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/livy-core_2.11-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-core_2.11-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/livy-repl_2.11-0.6.0-incubating.jar at spark://e54fbcdd1f92:38395/jars/livy-repl_2.11-0.6.0-incubating.jar with timestamp 1600078756322
20/09/14 10:19:16 INFO SparkContext: Added JAR file:///src/livy/apache-livy-0.6.0-incubating-bin/repl_2.11-jars/commons-codec-1.9.jar at spark://e54fbcdd1f92:38395/jars/commons-codec-1.9.jar with timestamp 1600078756322

I don't see uploaded jar file attached to spark context.

Is there anything wrong I am doing. 

In Batch its working fine.

Any help/suggestion will be highly appreciated.

Regards,
Badal

Don't have an account?
Coming from Hortonworks? Activate your account here