question Re: Spark - separating dependecies of spark and application in Support Questions

Spark - separating dependecies of spark and application

Jarinek — Thu, 13 May 2021 16:17:43 GMT

Hello,
I struggle on a dependency issues in spark, now. Being new in spark, I hope there is a simple remedy.
The question is, is there any mechanism how to separate dependencies of the spark engine and dependencies of a application.

Example: The latest version of spark-core_2.12 (3.1.1, March 2021) depends on
hadoop-client (3.3.0, March 2020) which itself depends on hadoop-common (3.3.0, July 2020)
which finally depends on an antient version of gson (2.2.4, May 2013).

You can easily find many other examples, e.g. commons-codec, protobuf-java ...

So, what if your application, basically a library developed outside spark, depends on the latest (no longer compatible) version of gson 2.8.6?

My obviously naive approach to start a spark application ends in runtime incompatibility clashes (e.g. with gson)

Best regards
Jaro

Re: Spark - separating dependecies of spark and application

mridley — Fri, 14 May 2021 14:41:37 GMT

Hello,

You can solve this by using the Maven shade plugin. Take a look at the Cloudera doc https://docs.cloudera.com/runtime/7.2.9/developing-spark-applications/topics/spark-packaging-different-versions-of-libraries-with-an-application.html .

Michael

Re: Spark - separating dependecies of spark and application

cjervis — Mon, 24 May 2021 12:47:55 GMT

Hi @Jarinek,

Did the reply from @mridley resolve your issue? If so, please mark the reply as the solution, as it will make it easier for others to find the answer in the future.