- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?
- Labels:
-
Apache Ambari
-
Apache Spark
-
Apache Zeppelin
Created 11-10-2016 10:30 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't have any problem to import external library for Spark Interpreter using SPARK_SUBMIT_OPTIONS.
This method doesn't work with Livy Interpreter.
What is the best solution to import external library for Livy Interpreter using zeppelin ?
I prefer to import from local JARs without having to use remote repositories.
Thank you in advance.
Created 11-11-2016 12:16 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can load dynamic library to livy interpreter by set livy.spark.jars.packages
property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.
Example
Property | Example | Description |
---|---|---|
livy.spark.jars.packages | io.spray:spray-json_2.10:1.3.1 | Adding extra libraries to livy interpreter |
https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/livy.html#adding-external-libraries
Created 11-10-2016 11:16 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@A. Karray You can specify JARs to use with Livy jobs using livy.spark.jars in the Livy interpreter conf. This should be a comma separated list of JAR locations which must be stored on HDFS. Currently local files cannot be used (i.e. they won't be localized on the cluster when the job runs.) It is a global setting so all JARs listed will be available for all Livy jobs run by all users.
Created 12-04-2016 05:53 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This solution doesn't work for me with yarn cluster mode configuration.
When I print sc.jars I can see that i have added the dependencies : hdfs:///user/zeppelin/lib/postgresql-9.4-1203-jdbc42.jar
But I's not possible to import any class of the Jar
<console>:30: error: object postgresql is not a member of package org import org.postgresql.Driver
Created 12-19-2016 03:46 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, did you find a solution?
I have the same pb ...
Thanks
Created 11-11-2016 12:16 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can load dynamic library to livy interpreter by set livy.spark.jars.packages
property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.
Example
Property | Example | Description |
---|---|---|
livy.spark.jars.packages | io.spray:spray-json_2.10:1.3.1 | Adding extra libraries to livy interpreter |
https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/livy.html#adding-external-libraries
Created 11-11-2016 03:27 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
This works fine for artifacts in maven central repository. Do you know if there is a way to define a custom maven remote repository?
I have tried using the livy.spark.jars.ivy according to the link below, but Livy still tries to retrieve the artifact from maven central.
http://spark.apache.org/docs/latest/configuration.html
Thanks!
Created 12-04-2016 05:48 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This solution doesn't work for me with yarn cluster mode configuration.
Created 12-05-2016 08:18 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi ,
did you find a solution to include libraries from internal maven repository?
When I inspect log files, I can see that livy tries to resolve dependencies with
http://dl.bintray.com/spark-packages, https://repo1.maven.org/, local-m2-cache.
Is there a way to add custom maven repository?
I'm using Ambari and Zeppelin
Thanks
Created 12-13-2016 02:22 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The jars should be able to be added by using the parameter key livy.spark.jars and pointing to an hdfs location in the livy interpreter settings. This does not seem to work. I had to place the needed jar in the following directory on the livy server:
/usr/hdp/2.5.3.0-37/livy/repl-jars
Created 12-13-2016 04:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ian ,
Thanks for your response, unfortunately it doesn't work.
I've added all jars in the /usr/hdp/current/livy-server/repl-jars folder.
In Spark environment I can see them with those properties:
- spark.yarn.secondary.jars
- spark.jars
All jars are present into the container folder :
hadoop/yarn/local/usercache/mgervais/appcache/application_1481623014483_0014/container_e24_1481623014483_0014_01_000001
I'm using Zeppelin, Livy & Spark. (Installed with Ambari.)
Any idea?
Thanks