spark's HikariCP-2.5.1.jar is eclipsing the desired HikariCP jar

[#1]

Hi,

There are two HikariCP jars in the class path:

1. /data/perftools/spark-prod-reports-3.0.0/jars/HikariCP-2.5.1.jar
2. /data/perftools/collector-spark-sandbox/lib/HikariCP-3.2.0.jar

The 1st is pre-packaged with spark, the 2nd is included by me.

Since /data/perftools/spark-prod-reports-3.0.0/bin/spark-submit's list of jars is checked first before checking my jar list, the classloader is loading HikariCP-2.5.1.jar instead of the desired one, HikariCP-3.2.0.jar that is provided via the flag  --jars . 

The problem is shown below. Notice the 4th and 5th lines are loading Hikari from the default spark package jar folder, not from my jar list located at /data/perftools/collector-spark-sandbox/lib/.

[Loaded slick.util.ConfigExtensionMethods$ from file:/data/perftools/collector-spark-sandbox/lib/slick_2.12-3.3.3.jar]
[Loaded slick.jdbc.JdbcDataSource$$$Lambda$1478/767152605 from slick.jdbc.JdbcDataSource$]
[Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$ from file:/data/perftools/collector-spark-sandbox/lib/slick-hikaricp_2.12-3.3.3.jar]
[Loaded com.zaxxer.hikari.HikariConfigMXBean from file:/data/perftools/spark-prod-reports-3.0.0/jars/HikariCP-2.5.1.jar]
[Loaded com.zaxxer.hikari.HikariConfig from file:/data/perftools/spark-prod-reports-3.0.0/jars/HikariCP-2.5.1.jar]
[Loaded slick.util.ConfigExtensionMethods$$$Lambda$1479/1522068461 from slick.util.ConfigExtensionMethods$]
[Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1480/60429889 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$]
[Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1481/788726348 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$]
[Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1482/945329680 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$]
[Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1483/1797498692 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$]
[Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1484/1543887118 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$]
[Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1485/1847812322 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$]
[Loaded slick.util.ConfigExtensionMethods$$$Lambda$1486/305632032 from slick.util.ConfigExtensionMethods$]

How can I get spark-submit to load my HikariCP-3.2.0.jar instead of the prepackaged one???

Thank you

[#2]

Hi @sparkdeveloper 

You are encountering the well-known phenomenon of Java Class shadowing, and it really doesn't have all that much to do with Spark, or the spark-submit utility. One direct solution is to employ a technique called "shading" the dependencies, which usually winds up being a lot of work, and I personally avoid doing it. If you want to go that route, you can read all about that technique on any number of websites that cover Java software development or in books on the same topic.

Obviously, I don't know what the code you're running that uses this library does, but if you're looking for some way to resolve this quickly, I think you are better off rewriting your code that uses the 3.2.0 version of this library to use the APIs in the version that comes pre-packaged with your spark installation and just omit adding your .jar file to the CLASSPATH. You'll probably need to download the HikariCP-2.5.1.jar file and install it where you're doing your development work. I realize that is a lazy kind of answer, but you should consider it unless you really have a compelling need to use the later version.

[#3]

Thanks @ask_bill_brooks . Much appreciate the reply.

I did solve the problem using an uber/fat jar. However, I was hoping for an elegant solution.  I did look into shading on gradle but it was too confusing and difficult to explain and maintain. sbt didn't seem to have shading (or an easier approaching to shading). 

At the moment, we are stuck with a fat jar. Hope there is an easier deployment method in the future via the dependencies.scala file.