Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Not able to run Java & Python livy examples

Not able to run Java & Python livy examples

New Contributor

Hi, I am new to spark (and Java + Scala also) and still trying to setup the standalone mode spark with livy job manager. I am not able to run the examples given in the livy example directory for any of the languages. I am sure I am missing something here, but not able to figure out what's missing. The examples are working properly if run from Spark direcoty.

  • I am using Ubuntu 16.04.3.
  • Other Installed versions :
    • 1. Spark Version : 2.2.1
  • 2. Java Version :
    • openjdk version "1.8.0_162"
    • OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12)
    • OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)
  • 3. Livy Version : 0.6.0-incubating-SNAPSHOT

Error in Java Example : 1) setting java classpaths :

JAVA_CP=<path>/spark-core_<version>.jar:<path>/livy-api-0.6.0-incubating-SNAPSHOT.jar:<path>/livy-client-http-0.6.0-incubating-SNAPSHOT.jar:<path>/livy-examples-0.6.0-incubating-SNAPSHOT.jar:<path>/slf4j-1.7.25/slf4j-api-1.7.25.jar:<path>/slf4j-1.7.25/slf4j-simple-1.7.25.jar

2) running example from livy home direcoty :

java -cp $JAVA_CP org.apache.livy.examples.PiApp http://localhost:8998 2

3) Error :

Uploading livy-example jar to the SparkContext... Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: Bad Request: "requirement failed: Local path /root/.livy-sessions/f2169a4c-41eb-4b1d-8b77-c9eaf853b949/livy-examples-0.6.0-incubating-SNAPSHOT.jar cannot be added to user sessions." at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.livy.examples.PiApp.main(PiApp.java:87) Caused by: java.io.IOException: Bad Request: "requirement failed: Local path /root/.livy-sessions/f2169a4c-41eb-4b1d-8b77-c9eaf853b949/livy-examples-0.6.0-incubating-SNAPSHOT.jar cannot be added to user sessions." at org.apache.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:229) at org.apache.livy.client.http.LivyConnection.post(LivyConnection.java:192) at org.apache.livy.client.http.HttpClient$2.call(HttpClient.java:152) at org.apache.livy.client.http.HttpClient$2.call(HttpClient.java:149) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

4) I have added the path /root/.livy-sessions/ in the whitelist config param but that did not solve the error. Error in Python example :

1) python ./examples/src/main/python/pi_app.py http://localhost:8998 2

2) Error in the execution :

Exception: org.apache.livy.shaded.kryo.kryo.KryoException: Encountered unregistered class ID: 254 org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656) org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767) org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63) org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39) org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27) org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57) org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42) org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

Can anybody please tell me what is missing here in both the examples?