question Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError in Support Questions

Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

kctan6 — Wed, 15 Jun 2016 10:32:47 GMT

Hi, I'm experimenting with using Zeppelin / Spark to perform geo-location on IP addresses using the Maxmind GeoIP library. I am encountering a NoSuchMethodError which from reading the forums appears to be a dependency issue with the method not being in certain versions of the jackson lib. How can I go about identifying and resolving this dependency issue in Zeppelin? I load geoip2 via %dep and have removed the older versions of the jackson lib from zeppelin/lib/lib to no avail. Thanks!

%dep
z.addRepo("geoip2").url("http://mvnrepository.com/artifact/com.maxmind.geoip2/geoip2/2.7.0")
z.load("com.maxmind.geoip2:geoip2:2.7.0")

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

rajkumar_singh — Wed, 15 Jun 2016 12:30:24 GMT

seems you have library mismatch for jackson, could you please verify which jars contain this class com.fasterxml.jackson.databind.node.ArrayNode,isolate those jars and maintain a single version of jars available across the classpath. if you still want to use to different jar version then please consider jar shading.

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

kctan6 — Wed, 15 Jun 2016 12:52:57 GMT

Hi @Rajkumar Singh, I believe the jar in question is jackson-databind. For Spark / Zeppelin which classpath / directories should I be look at?

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

rajkumar_singh — Wed, 15 Jun 2016 12:58:55 GMT

please look into zeepline lib folder.

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

kctan6 — Wed, 15 Jun 2016 13:25:41 GMT

Yeah have tried that but still encountering the same error

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

rajkumar_singh — Wed, 15 Jun 2016 14:10:03 GMT

@KCcan you lookout for these two jars, i m suspecting that zepplien spark dependency is causing this issue.

zeppelin/lib/interpreter/spark/dep/zeppelin-spark-dependencies-0.6.0.2.4.0.0-169.jar

/zeppelin/lib/interpreter/flink/jackson-databind-2.4.2.jar

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

kctan6 — Wed, 15 Jun 2016 14:30:44 GMT

@Rajkumar Singh yeah i removed the jackson-databind jar in the flink folder, the zeppelin-spark-dependencies jar is still there. How do I determine which version / jar is being used by Zeppelin / Spark at run-time?

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

rajkumar_singh — Wed, 15 Jun 2016 14:44:29 GMT

this jar is used by the zeppelin server at runtime zeppelin/lib/interpreter/spark/dep/zeppelin-spark-dependencies-0.6.0.2.4.0.0-169.jar, will be good if you have your version of jar shaded.

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

kctan6 — Thu, 16 Jun 2016 08:35:11 GMT

thanks @Rajkumar Singh for the suggestion. are you able to provide some instructions or direct me to some resources on how I should shade the jar?

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

rajkumar_singh — Thu, 16 Jun 2016 11:34:37 GMT

@KC could you please follow this tutorial http://www.mkyong.com/maven/create-a-fat-jar-file-maven-shade-plugin/

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

phargis — Fri, 17 Jun 2016 01:48:35 GMT

Spark includes some Jackson libraries as it's own dependencies, including this one:

   <fasterxml.jackson.version>2.6.5</fasterxml.jackson.version>

Therefore, if your additional third-party library also includes this library with a different version, then the classloader will get errors. You can use the Maven Shade plugin to "relocate" the third-party jar, as described here:

https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html

Here is an example of relocating the "com.fasterxml.jackson" library:

http://stackoverflow.com/questions/34764732/relocating-fastxml-jackson-classes-to-my-package-fastxml-jackson

Re: Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

kctan6 — Fri, 17 Jun 2016 14:04:23 GMT

Thanks! Finally got it working