Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Zeppelin Spark Maxmind jackson.databind NoSuchMethodError

avatar
Rising Star

Hi, I'm experimenting with using Zeppelin / Spark to perform geo-location on IP addresses using the Maxmind GeoIP library. I am encountering a NoSuchMethodError which from reading the forums appears to be a dependency issue with the method not being in certain versions of the jackson lib. How can I go about identifying and resolving this dependency issue in Zeppelin? I load geoip2 via %dep and have removed the older versions of the jackson lib from zeppelin/lib/lib to no avail. Thanks!

%dep
z.addRepo("geoip2").url("http://mvnrepository.com/artifact/com.maxmind.geoip2/geoip2/2.7.0")
z.load("com.maxmind.geoip2:geoip2:2.7.0")
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
1 ACCEPTED SOLUTION

avatar

Spark includes some Jackson libraries as it's own dependencies, including this one:

   <fasterxml.jackson.version>2.6.5</fasterxml.jackson.version>

Therefore, if your additional third-party library also includes this library with a different version, then the classloader will get errors. You can use the Maven Shade plugin to "relocate" the third-party jar, as described here:

https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html

Here is an example of relocating the "com.fasterxml.jackson" library:

http://stackoverflow.com/questions/34764732/relocating-fastxml-jackson-classes-to-my-package-fastxml...

View solution in original post

11 REPLIES 11

avatar
Super Guru

seems you have library mismatch for jackson, could you please verify which jars contain this class com.fasterxml.jackson.databind.node.ArrayNode,isolate those jars and maintain a single version of jars available across the classpath. if you still want to use to different jar version then please consider jar shading.

avatar
Rising Star

Hi @Rajkumar Singh, I believe the jar in question is jackson-databind. For Spark / Zeppelin which classpath / directories should I be look at?

avatar
Super Guru

please look into zeepline lib folder.

avatar
Rising Star

Yeah have tried that but still encountering the same error

avatar
Super Guru

@KCcan you lookout for these two jars, i m suspecting that zepplien spark dependency is causing this issue.

zeppelin/lib/interpreter/spark/dep/zeppelin-spark-dependencies-0.6.0.2.4.0.0-169.jar

/zeppelin/lib/interpreter/flink/jackson-databind-2.4.2.jar

avatar
Rising Star

@Rajkumar Singh yeah i removed the jackson-databind jar in the flink folder, the zeppelin-spark-dependencies jar is still there. How do I determine which version / jar is being used by Zeppelin / Spark at run-time?

avatar
Super Guru

this jar is used by the zeppelin server at runtime zeppelin/lib/interpreter/spark/dep/zeppelin-spark-dependencies-0.6.0.2.4.0.0-169.jar, will be good if you have your version of jar shaded.

avatar
Rising Star

thanks @Rajkumar Singh for the suggestion. are you able to provide some instructions or direct me to some resources on how I should shade the jar?

avatar
Super Guru