oozie share lib for oozie contains a lot of jar files in 2.3.4. why?

[#1]

Hi, we recently upgraded from HDP 2.2.X to 2.3.4 and I notice that /user/oozie/share/lib/oozie contains a lot more jar files now. See below. The presence of these jar files is puzzling since they are otherwise in the classpath of, say, java oozie actions. e.g. in directories like `mr-framework/hadoop/share/hadoop/common/lib/`

Does anyone know 1) why these jars in share/lib/oozie and 2) if they can be safely removed? Thanks!

2.3.4:

-rw-r--r--   3 oozie hadoop   11948376 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/aws-java-sdk-1.7.4.jar
-rw-r--r--   3 oozie hadoop     666763 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/azure-storage-2.2.0.jar
-rw-r--r--   3 oozie hadoop     412739 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/commons-lang3-3.3.2.jar
-rw-r--r--   3 oozie hadoop    1648200 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/guava-11.0.2.jar
-rw-r--r--   3 oozie hadoop     103052 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/hadoop-aws-2.7.1.2.3.4.0-3485.jar
-rw-r--r--   3 oozie hadoop     137439 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/hadoop-azure-2.7.1.2.3.4.0-3485.jar
-rw-r--r--   3 oozie hadoop      33483 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-annotations-2.2.3.jar
-rw-r--r--   3 oozie hadoop     192699 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-core-2.2.3.jar
-rw-r--r--   3 oozie hadoop     865838 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/jackson-databind-2.2.3.jar
-rw-r--r--   3 oozie hadoop     570478 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/joda-time-2.1.jar
-rw-r--r--   3 oozie hadoop      16046 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/json-simple-1.1.jar
-rw-r--r--   3 oozie hadoop      12294 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/oozie-hadoop-utils-hadoop-2-4.2.0.2.3.4.0-3485.jar
-rw-r--r--   3 oozie hadoop      49474 2016-02-17 15:19 /user/oozie/share/lib/lib_20160217151921/oozie/oozie-sharelib-oozie-4.2.0.2.3.4.0-3485.jar

2.2

-rw-r--r--   3 hdfs  hadoop       7832 2015-09-02 12:57 /user/oozie/share/lib/oozie/core-site.xml
-rw-r--r--   3 hdfs  hadoop       9517 2015-09-02 12:57 /user/oozie/share/lib/oozie/hdfs-site.xml
-rwxr-xr-x   3 oozie hadoop      16046 2015-06-17 23:08 /user/oozie/share/lib/oozie/json-simple-1.1.jar
-rw-r--r--   3 hdfs  hadoop       6671 2015-09-02 12:57 /user/oozie/share/lib/oozie/mapred-site.xml
-rwxr-xr-x   3 oozie hadoop       3242 2015-06-17 23:08 /user/oozie/share/lib/oozie/oozie-hadoop-utils-2.6.0.2.2.4.12-1.oozie-4.1.0.2.2.4.12-1.jar
-rwxr-xr-x   3 oozie hadoop      41903 2015-06-17 23:08 /user/oozie/share/lib/oozie/oozie-sharelib-oozie-4.1.0.2.2.4.12-1.jar
-rw-r--r--   3 hdfs  hadoop      18164 2015-11-04 18:55 /user/oozie/share/lib/oozie/yarn-site.xml

[#2 · Accepted solution]

@Aaron Dossett

Please don't remove these JARs 🙂 Yes its by design to support more functionality and fix existing bugs.

[#3]

Thank you @Kuldeep Kulkarni! It's causing me some problems because files like jackson-*.jar and guava appear early in the classpath (the current directory of the oozie action) and late in the classpath (mr-framework/hadoop/share/hadoop/common/lib/). I would effectively have to pin the dependencies of my java action to those precise versions. Any way around that?

[#4]

you can explicitly set your dependencies in your java program and exclude dependencies for the version causing you problems.

[#5]

@Aaron Dossett - Can you please check below comment by Artem

[#6]

Thanks @Artem Ervits and @Kuldeep Kulkarni. Yes, I can definitely control the versions that my java application includes. The problem is that when the java action runs, the jar files that I created are in the "current directory" of the java action along with the other versions that oozie share lib is including. These jar files have the same classpath precedence so it appears to be random which jar is picked up. For example, com.fasterxml.jackson.core.JsonFactory might be found in my own jar or in jackson-core-2.2.3.jar provided by oozie.

[#7]

@Aaron Dossett - Okay if that is the case, can you please try removing conflicting jar file(s) from oozie sharelib and see if that helps. If you want to revert the changes you can just re-create oozie sharelib by using below command

Note - Please run below commands on oozie host by oozie user.

/usr/hdp/2.3.2.0-2950/oozie/bin/oozie-setup.sh sharelib create -locallib /usr/hdp/<version>/oozie/oozie-sharelib.tar.gz -fs hdfs://<namenode-host>:8020

oozie admin -oozie http://<oozie-host>:11000/oozie -sharelibupdate

[#8]

Thanks @Kuldeep Kulkarni. Since the cluster I'm on is shared by many users for many different workloads I won't be able to make that change easily. Are there oozie JIRAs regarding this change I could read or was this specific to HWX packaging? I'd like to learn more about it since it seems very disruptive to java actions, e.g. having a 4+ year old version of guava in the classpath. Thanks!

[#9]

I was in a similar situation, to my surprise, the issue was on the client side, my .M2 directory had old version of jackson that was being appended before the proper version, I ended up using exclude tags

[#10]

I had the same problem that dependencies got into conflict. I used Maven shade plugin to relocate all my dependencies.

https://maven.apache.org/components/plugins/maven-shade-plugin/examples/class-relocation.html