How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

[#1]

I don't have any problem to import external library for Spark Interpreter using SPARK_SUBMIT_OPTIONS.

This method doesn't work with Livy Interpreter.

What is the best solution to import external library for Livy Interpreter using zeppelin ?

I prefer to import from local JARs without having to use remote repositories.

Thank you in advance.

[#2]

@A. Karray You can specify JARs to use with Livy jobs using livy.spark.jars in the Livy interpreter conf. This should be a comma separated list of JAR locations which must be stored on HDFS. Currently local files cannot be used (i.e. they won't be localized on the cluster when the job runs.) It is a global setting so all JARs listed will be available for all Livy jobs run by all users.

[#3]

This solution doesn't work for me with yarn cluster mode configuration.

When I print sc.jars I can see that i have added the dependencies : hdfs:///user/zeppelin/lib/postgresql-9.4-1203-jdbc42.jar

But I's not possible to import any class of the Jar

<console>:30: error: object postgresql is not a member of package org import org.postgresql.Driver

[#4]

Hi, did you find a solution?

I have the same pb ...

Thanks

[#5 · Accepted solution]

You can load dynamic library to livy interpreter by set livy.spark.jars.packages property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.

Example

Property	Example	Description
livy.spark.jars.packages	io.spray:spray-json_2.10:1.3.1	Adding extra libraries to livy interpreter

https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/livy.html#adding-external-libraries

[#6]

Hi,

This works fine for artifacts in maven central repository. Do you know if there is a way to define a custom maven remote repository?

I have tried using the livy.spark.jars.ivy according to the link below, but Livy still tries to retrieve the artifact from maven central.

http://spark.apache.org/docs/latest/configuration.html

Thanks!

[#7]

This solution doesn't work for me with yarn cluster mode configuration.

[#8]

Hi ,

did you find a solution to include libraries from internal maven repository?

When I inspect log files, I can see that livy tries to resolve dependencies with

http://dl.bintray.com/spark-packages, https://repo1.maven.org/, local-m2-cache.

Is there a way to add custom maven repository?

I'm using Ambari and Zeppelin

Thanks

[#9]

The jars should be able to be added by using the parameter key livy.spark.jars and pointing to an hdfs location in the livy interpreter settings. This does not seem to work. I had to place the needed jar in the following directory on the livy server:

/usr/hdp/2.5.3.0-37/livy/repl-jars

[#10]

Hi Ian ,

Thanks for your response, unfortunately it doesn't work.

I've added all jars in the /usr/hdp/current/livy-server/repl-jars folder.

In Spark environment I can see them with those properties:

• spark.yarn.secondary.jars
• spark.jars

All jars are present into the container folder :

hadoop/yarn/local/usercache/mgervais/appcache/application_1481623014483_0014/container_e24_1481623014483_0014_01_000001

I'm using Zeppelin, Livy & Spark. (Installed with Ambari.)

Any idea?

Thanks