Cloudera Community
sparkdeveloper
user rank
New Contributor

Re: spark's HikariCP-2.5.1.jar is eclipsing the de...

by New Contributor in Support Questions
‎05-08-2022 07:05 PM
‎05-08-2022 07:05 PM
Thanks @ask_bill_brooks . Much appreciate the reply.   I did solve the problem using an uber/fat jar. However, I was hoping for an elegant solution.  I did look into shading on gradle but it was too confusing and difficult to explain and maintain. sbt didn't seem to have shading (or an easier approaching to shading).  At the moment, we are stuck with a fat jar. Hope there is an easier deployment method in the future via the dependencies.scala file. ... View more

spark's HikariCP-2.5.1.jar is eclipsing the desire...

by New Contributor in Support Questions
‎04-29-2022 10:08 PM
‎04-29-2022 10:08 PM
  Hi, There are two HikariCP jars in the class path: /data/perftools/spark-prod-reports-3.0.0/jars/HikariCP-2.5.1.jar /data/perftools/collector-spark-sandbox/lib/HikariCP-3.2.0.jar The 1st is pre-packaged with spark, the 2nd is included by me.   Since /data/perftools/spark-prod-reports-3.0.0/bin/spark-submit's list of jars is checked first before checking my jar list, the classloader is loading HikariCP-2.5.1.jar instead of the desired one, HikariCP-3.2.0.jar that is provided via the flag  --jars .    The problem is shown below. Notice the 4th and 5th lines are loading Hikari from the default spark package jar folder, not from my jar list located at /data/perftools/collector-spark-sandbox/lib/. [Loaded slick.util.ConfigExtensionMethods$ from file:/data/perftools/collector-spark-sandbox/lib/slick_2.12-3.3.3.jar] [Loaded slick.jdbc.JdbcDataSource$$$Lambda$1478/767152605 from slick.jdbc.JdbcDataSource$] [Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$ from file:/data/perftools/collector-spark-sandbox/lib/slick-hikaricp_2.12-3.3.3.jar] [Loaded com.zaxxer.hikari.HikariConfigMXBean from file:/data/perftools/spark-prod-reports-3.0.0/jars/HikariCP-2.5.1.jar] [Loaded com.zaxxer.hikari.HikariConfig from file:/data/perftools/spark-prod-reports-3.0.0/jars/HikariCP-2.5.1.jar] [Loaded slick.util.ConfigExtensionMethods$$$Lambda$1479/1522068461 from slick.util.ConfigExtensionMethods$] [Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1480/60429889 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$] [Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1481/788726348 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$] [Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1482/945329680 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$] [Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1483/1797498692 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$] [Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1484/1543887118 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$] [Loaded slick.jdbc.hikaricp.HikariCPJdbcDataSource$$$Lambda$1485/1847812322 from slick.jdbc.hikaricp.HikariCPJdbcDataSource$] [Loaded slick.util.ConfigExtensionMethods$$$Lambda$1486/305632032 from slick.util.ConfigExtensionMethods$]   How can I get spark-submit to load my HikariCP-3.2.0.jar instead of the prepackaged one???   Thank you ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎05-08-2022 10:26 PM
Community Statistics
Member Since ‎04-29-2022 09:55 PM
Last Visited ‎05-08-2022 10:26 PM
Posts 2