Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Not able to run Java & Python livy examples


Not able to run Java & Python livy examples

New Contributor

Hi, I am new to spark (and Java + Scala also) and still trying to setup the standalone mode spark with livy job manager. I am not able to run the examples given in the livy example directory for any of the languages. I am sure I am missing something here, but not able to figure out what's missing. The examples are working properly if run from Spark direcoty.

  • I am using Ubuntu 16.04.3.
  • Other Installed versions :
    • 1. Spark Version : 2.2.1
  • 2. Java Version :
    • openjdk version "1.8.0_162"
    • OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12)
    • OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)
  • 3. Livy Version : 0.6.0-incubating-SNAPSHOT

Error in Java Example : 1) setting java classpaths :


2) running example from livy home direcoty :

java -cp $JAVA_CP org.apache.livy.examples.PiApp http://localhost:8998 2

3) Error :

Uploading livy-example jar to the SparkContext... Exception in thread "main" java.util.concurrent.ExecutionException: Bad Request: "requirement failed: Local path /root/.livy-sessions/f2169a4c-41eb-4b1d-8b77-c9eaf853b949/livy-examples-0.6.0-incubating-SNAPSHOT.jar cannot be added to user sessions." at at java.util.concurrent.FutureTask.get( at org.apache.livy.examples.PiApp.main( Caused by: Bad Request: "requirement failed: Local path /root/.livy-sessions/f2169a4c-41eb-4b1d-8b77-c9eaf853b949/livy-examples-0.6.0-incubating-SNAPSHOT.jar cannot be added to user sessions." at org.apache.livy.client.http.LivyConnection.sendRequest( at at org.apache.livy.client.http.HttpClient$ at org.apache.livy.client.http.HttpClient$ at at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201( at java.util.concurrent.ScheduledThreadPoolExecutor$ at java.util.concurrent.ThreadPoolExecutor.runWorker( at java.util.concurrent.ThreadPoolExecutor$ at

4) I have added the path /root/.livy-sessions/ in the whitelist config param but that did not solve the error. Error in Python example :

1) python ./examples/src/main/python/ http://localhost:8998 2

2) Error in the execution :

Exception: org.apache.livy.shaded.kryo.kryo.KryoException: Encountered unregistered class ID: 254 org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass( org.apache.livy.shaded.kryo.kryo.Kryo.readClass( org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject( org.apache.livy.client.common.Serializer.deserialize( java.util.concurrent.ThreadPoolExecutor.runWorker( java.util.concurrent.ThreadPoolExecutor$

Can anybody please tell me what is missing here in both the examples?

Don't have an account?
Coming from Hortonworks? Activate your account here