Created 06-15-2016 03:32 AM
Hi, I'm experimenting with using Zeppelin / Spark to perform geo-location on IP addresses using the Maxmind GeoIP library. I am encountering a NoSuchMethodError which from reading the forums appears to be a dependency issue with the method not being in certain versions of the jackson lib. How can I go about identifying and resolving this dependency issue in Zeppelin? I load geoip2 via %dep and have removed the older versions of the jackson lib from zeppelin/lib/lib to no avail. Thanks!
%dep z.addRepo("geoip2").url("http://mvnrepository.com/artifact/com.maxmind.geoip2/geoip2/2.7.0") z.load("com.maxmind.geoip2:geoip2:2.7.0")
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ArrayNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/List;)V
Created 06-16-2016 06:48 PM
Spark includes some Jackson libraries as it's own dependencies, including this one:
<fasterxml.jackson.version>2.6.5</fasterxml.jackson.version>
Therefore, if your additional third-party library also includes this library with a different version, then the classloader will get errors. You can use the Maven Shade plugin to "relocate" the third-party jar, as described here:
https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
Here is an example of relocating the "com.fasterxml.jackson" library:
Created 06-17-2016 07:04 AM
Thanks! Finally got it working
Created 06-16-2016 06:48 PM
Spark includes some Jackson libraries as it's own dependencies, including this one:
<fasterxml.jackson.version>2.6.5</fasterxml.jackson.version>
Therefore, if your additional third-party library also includes this library with a different version, then the classloader will get errors. You can use the Maven Shade plugin to "relocate" the third-party jar, as described here:
https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html
Here is an example of relocating the "com.fasterxml.jackson" library: