question Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ? in Archives of Support Questions (Read Only)

How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

karray_ali — Thu, 10 Nov 2016 18:30:08 GMT

I don't have any problem to import external library for Spark Interpreter using SPARK_SUBMIT_OPTIONS.

This method doesn't work with Livy Interpreter.

What is the best solution to import external library for Livy Interpreter using zeppelin ?

I prefer to import from local JARs without having to use remote repositories.

Thank you in advance.

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

tstebbens — Thu, 10 Nov 2016 19:16:06 GMT

@A. Karray You can specify JARs to use with Livy jobs using livy.spark.jars in the Livy interpreter conf. This should be a comma separated list of JAR locations which must be stored on HDFS. Currently local files cannot be used (i.e. they won't be localized on the cluster when the job runs.) It is a global setting so all JARs listed will be available for all Livy jobs run by all users.

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

azeltov — Fri, 11 Nov 2016 08:16:55 GMT

You can load dynamic library to livy interpreter by set livy.spark.jars.packages property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.

Example

Property	Example	Description
livy.spark.jars.packages	io.spray:spray-json_2.10:1.3.1	Adding extra libraries to livy interpreter

https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/livy.html#adding-external-libraries

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

tkak — Fri, 11 Nov 2016 23:27:17 GMT

Hi,

This works fine for artifacts in maven central repository. Do you know if there is a way to define a custom maven remote repository?

I have tried using the livy.spark.jars.ivy according to the link below, but Livy still tries to retrieve the artifact from maven central.

http://spark.apache.org/docs/latest/configuration.html

Thanks!

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

karray_ali — Mon, 05 Dec 2016 01:48:45 GMT

This solution doesn't work for me with yarn cluster mode configuration.

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

karray_ali — Mon, 05 Dec 2016 01:53:04 GMT

This solution doesn't work for me with yarn cluster mode configuration.

When I print sc.jars I can see that i have added the dependencies : hdfs:///user/zeppelin/lib/postgresql-9.4-1203-jdbc42.jar

But I's not possible to import any class of the Jar

<console>:30: error: object postgresql is not a member of package org import org.postgresql.Driver

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

mgervais — Mon, 05 Dec 2016 16:18:51 GMT

Hi ,

did you find a solution to include libraries from internal maven repository?

When I inspect log files, I can see that livy tries to resolve dependencies with

http://dl.bintray.com/spark-packages, https://repo1.maven.org/, local-m2-cache.

Is there a way to add custom maven repository?

I'm using Ambari and Zeppelin

Thanks

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

iroberts — Tue, 13 Dec 2016 22:22:53 GMT

The jars should be able to be added by using the parameter key livy.spark.jars and pointing to an hdfs location in the livy interpreter settings. This does not seem to work. I had to place the needed jar in the following directory on the livy server:

/usr/hdp/2.5.3.0-37/livy/repl-jars

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

mgervais — Wed, 14 Dec 2016 00:21:46 GMT

Hi Ian ,

Thanks for your response, unfortunately it doesn't work.

I've added all jars in the /usr/hdp/current/livy-server/repl-jars folder.

In Spark environment I can see them with those properties:

spark.yarn.secondary.jars
spark.jars

All jars are present into the container folder :

hadoop/yarn/local/usercache/mgervais/appcache/application_1481623014483_0014/container_e24_1481623014483_0014_01_000001

I'm using Zeppelin, Livy & Spark. (Installed with Ambari.)

Any idea?

Thanks

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

iroberts — Wed, 14 Dec 2016 00:25:06 GMT

@Mickaël GERVAIS check to make sure livy interpreter is listed in the interpreter bindings for the notebook. Also, set DEBUG on the livy server and check in the livy out file produced on the server. Finally, make sure you have restarted livy and zeppelin to pick up the changes. I tested and it did work for me.

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

mgervais — Wed, 14 Dec 2016 01:17:06 GMT

@Ian Roberts I'm sorry , but I cannot make it work.

Here is my error:

import org.apache.commons.codec.binary.Base64
import java.time.LocalDateTime.now
import com.mongodb.BasicDBObjectBuilder.start

import org.apache.commons.codec.binary.Base64
import java.time.LocalDateTime.now
<console>:35: error: object mongodb is not a member of package com
         import com.mongodb.BasicDBObjectBuilder.start

The jar mongo-java-driver-2.14.3.jar is present in:

livy-server/repl-jars
hdfs:///user/mgervais/.sparkStaging/application_1481647493263_0001

And the Spark UI show those properties:

spark.yarn.secondary.jars : ...mongo-java-driver-2.14.3.jar...
spark.jars : ...file:/usr/hdp/current/livy-server/repl-jars/mongo-java-driver-2.14.3.jar...

I've the livy interpreter enable in the Zeppelin notebook (It works , I can see the livy sessions...)

I've restarted the full Cluster with Ambari...

Thanks...

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

iroberts — Wed, 14 Dec 2016 01:27:35 GMT

This does not look like an issue with the jar being included but rather an issue with the import statement. I breifly looked on google and see similar descriptions stating to try org.mongodb. I would focus on the import statement more than the inclusion of the jar for livy.

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

mgervais — Wed, 14 Dec 2016 05:07:14 GMT

Okay, but I've the same issue with others imports which are not parts of native libraries.

My own jar cannot be included neither....

Is this a problem with Zeppelin notebook?

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

mgervais — Mon, 19 Dec 2016 18:51:41 GMT

Hi,

I'm using this version of HDP : /usr/hdp/2.5.0.0-1245/

Is this the reason? Should I upgrade my stack?

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

mgervais — Mon, 19 Dec 2016 23:46:39 GMT

Hi, did you find a solution?

I have the same pb ...

Thanks

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

tkak — Tue, 30 May 2017 03:34:27 GMT

Hi all,

Has anyone succeeded in using Livy with a custom maven repository?

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

Rachna_Dhand — Wed, 31 May 2017 15:03:00 GMT

hello @Laurence Da Luz

I am trying to run CRAN packages on Notebook. I downloaded the packages, and installed them on server, but somehow Zeppelin Notebook does not picks the package, for instance

%livy.sparkr

library(data.table)

returns in error on notebook.

Hope you can assist in that.

Rachna

Re: How to import External Libraries for Livy Interpreter using zeppelin (Using Yarn cluser mode) ?

james_jones — Wed, 21 Feb 2018 02:03:29 GMT

@Rachna Dhand I know you must be way past this issue, but -- You have to install the packages on all NodeManager nodes as root so they are available to all users. Maybe this will help someone else in the future.