Spark - separating dependecies of spark and application

[#1]

Hello,
I struggle on a dependency issues in spark, now. Being new in spark, I hope there is a simple remedy.
The question is, is there any mechanism how to separate dependencies of the spark engine and dependencies of a application.

Example: The latest version of spark-core_2.12 (3.1.1, March 2021) depends on
hadoop-client (3.3.0, March 2020) which itself depends on hadoop-common (3.3.0, July 2020)
which finally depends on an antient version of gson (2.2.4, May 2013).

You can easily find many other examples, e.g. commons-codec, protobuf-java ...

So, what if your application, basically a library developed outside spark, depends on the latest (no longer compatible) version of gson 2.8.6?

My obviously naive approach to start a spark application ends in runtime incompatibility clashes (e.g. with gson)

Best regards
Jaro

[#2 · Accepted solution]

Hello,

You can solve this by using the Maven shade plugin. Take a look at the Cloudera doc https://docs.cloudera.com/runtime/7.2.9/developing-spark-applications/topics/spark-packaging-differe... .

Michael

[#3]

Hi @Jarinek,

Did the reply from @mridley resolve your issue? If so, please mark the reply as the solution, as it will make it easier for others to find the answer in the future.