Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

avatar
Explorer

I don't have any problem to import external library for Spark Interpreter using SPARK_SUBMIT_OPTIONS.

This method doesn't work with Livy Interpreter.

What is the best solution to import external library for Livy Interpreter using zeppelin ?

I prefer to import from local JARs without having to use remote repositories.

Thank you in advance.

1 ACCEPTED SOLUTION

avatar

You can load dynamic library to livy interpreter by set livy.spark.jars.packages property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.

Example

PropertyExampleDescription
livy.spark.jars.packagesio.spray:spray-json_2.10:1.3.1Adding extra libraries to livy interpreter

https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/livy.html#adding-external-libraries

View solution in original post

17 REPLIES 17

avatar
Expert Contributor

@A. Karray You can specify JARs to use with Livy jobs using livy.spark.jars in the Livy interpreter conf. This should be a comma separated list of JAR locations which must be stored on HDFS. Currently local files cannot be used (i.e. they won't be localized on the cluster when the job runs.) It is a global setting so all JARs listed will be available for all Livy jobs run by all users.

avatar
Explorer

This solution doesn't work for me with yarn cluster mode configuration.

When I print sc.jars I can see that i have added the dependencies : hdfs:///user/zeppelin/lib/postgresql-9.4-1203-jdbc42.jar

But I's not possible to import any class of the Jar

<console>:30: error: object postgresql is not a member of package org import org.postgresql.Driver

avatar
Explorer

Hi, did you find a solution?

I have the same pb ...

Thanks

avatar

You can load dynamic library to livy interpreter by set livy.spark.jars.packages property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.

Example

PropertyExampleDescription
livy.spark.jars.packagesio.spray:spray-json_2.10:1.3.1Adding extra libraries to livy interpreter

https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/livy.html#adding-external-libraries

avatar
New Contributor

Hi,

This works fine for artifacts in maven central repository. Do you know if there is a way to define a custom maven remote repository?

I have tried using the livy.spark.jars.ivy according to the link below, but Livy still tries to retrieve the artifact from maven central.

http://spark.apache.org/docs/latest/configuration.html

Thanks!

avatar
Explorer

This solution doesn't work for me with yarn cluster mode configuration.

avatar
Explorer

Hi ,

did you find a solution to include libraries from internal maven repository?

When I inspect log files, I can see that livy tries to resolve dependencies with

http://dl.bintray.com/spark-packages, https://repo1.maven.org/, local-m2-cache.

Is there a way to add custom maven repository?

I'm using Ambari and Zeppelin

Thanks

avatar
Expert Contributor

The jars should be able to be added by using the parameter key livy.spark.jars and pointing to an hdfs location in the livy interpreter settings. This does not seem to work. I had to place the needed jar in the following directory on the livy server:

/usr/hdp/2.5.3.0-37/livy/repl-jars

avatar
Explorer

Hi Ian ,

Thanks for your response, unfortunately it doesn't work.

I've added all jars in the /usr/hdp/current/livy-server/repl-jars folder.

In Spark environment I can see them with those properties:

  • spark.yarn.secondary.jars
  • spark.jars

All jars are present into the container folder :

hadoop/yarn/local/usercache/mgervais/appcache/application_1481623014483_0014/container_e24_1481623014483_0014_01_000001

I'm using Zeppelin, Livy & Spark. (Installed with Ambari.)

Any idea?

Thanks