Support Questions
Find answers, ask questions, and share your expertise

Oozie overrides dependencies with shared libs,Problem with application dependencies being overridden by oozie shared libs

Highlighted

Oozie overrides dependencies with shared libs,Problem with application dependencies being overridden by oozie shared libs

New Contributor

Summary : I have an application that's configured with oozie java aciton. The problem is sometimes jars from oozie shared lib folders are loaded and my job fails

Here's the snipped of how workflow looks like

<java> <job-tracker>${jobTracker} </job-tracker> <name-node>${nameNode}</name-node>

<configuration>

<property>

<name>oozie.launcher.mapreduce.user.classpath.first</name>

<value>true</value>

</property> <property> <name>oozie.launcher.mapreduce.map.java.opts</name> <value>-verbose</value> </property> </configuration> <main-class>Myapp</main-class>

Myapp has a dependency on play-json (2.11-2.5.13) which in turn has a dependency on jackson jars for e.g. jackson-databind-2.7.8.jar . I provide these jars in my application lib folder ie. <workflowfolder>/lib/jackson-databind-2.7.8.jar

However, when I see the logs I see that sometimes oozie loads a different version of jackson-databind (I see these logs because I have enabled oozie.launcher.mapreduce.map.java.opts=-verbose

ie. [Loaded com.fasterxml.jackson.databind.ser.SerializerFactory from file:/disk/4/yarn/local/filecache/2878/jackson-databind-2.2.3.jar]

It finds this library because its in oozie shared lib folder 1. /user/oozie/share/lib/lib_20170315175040/oozie/jackson-databind-2.2.3.jar

I have tried to reset the following variables 1. oozie.use.system.libpath=false 2. oozie.launcher.mapreduce.user.classpath.first=true (This as a property in my workflow)

However still a few times my version of databind is loaded(2.7.8) and other times 2.2.3 is loaded which is when my application fails. How can I tell oozie to stop loading shared libs ?

,

Summary : Oozie java action not prioritizing provided jars over oozie shared lib

Oozie version : 4.2.0.2.3.6.2-3

Snipped for java action : <java>

<job-tracker>${jobTracker}</job-tracker>

<name-node>${nameNode}</name-node> <configuration>

<property> <name>oozie.launcher.mapreduce.user.classpath.first</name> <value>true</value> </property> <property> <name>oozie.launcher.mapreduce.map.java.opts</name> <value>-verbose</value> </property> </configuration>

<main-class>MyApp</main-class>

<arg>${bcosAction}</arg>

</java>

In my properties file I have set oozie.use.system.libpath=false

Myapp has a dependency on play-json_2.11-2.5.13.jar which in turn depend on jackson-databind-2.7.8.jar that I deploy in my lib folder for e.g. <workflow folder>/lib/jackson-databind-2.7.8.jar

However when I trigger the job I note the following in logs (By virtue of oozie.launcher.mapreduce.map.java.opts=-verbose)

[Loaded com.fasterxml.jackson.databind.type.ClassKey from file:/disk/4/yarn/local/filecache/2878/jackson-databind-2.2.3.jar]

This is because the databind library is part of oozie shared libs For e.g. 1. /user/oozie/share/lib/lib_20170315175040/oozie/jackson-databind-2.2.3.jar 2. /user/oozie/share/lib/lib_20170315175040/distcp/jackson-databind-2.2.3.jar

Quite a few times , the application loads my library jackson-databind-2.7.8.jar, however it fails once in a while.

How can I tell oozie to not use its shared libs ?

2 REPLIES 2

Re: Oozie overrides dependencies with shared libs,Problem with application dependencies being overridden by oozie shared libs

Explorer

Could you try add below four properties to your workflow corresponding java action ?

<name>mapreduce.job.user.classpath.first</name>

<value>true</value>

</property>

<property>

<name>mapreduce.task.classpath.user.precedence</name>

<value>true</value>

</property>

<property>

<name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>

<value>true</value>

</property>

<property>

<name>oozie.launcher.mapreduce.job.user.classpath.first</name>

<value>true</value>

</property>

Highlighted

Re: Oozie overrides dependencies with shared libs,Problem with application dependencies being overridden by oozie shared libs

New Contributor

Thanks, however I still see the same problem , now I have all the following in my java action <configuration> <property> <name>oozie.launcher.mapreduce.user.classpath.first</name> <value>true</value>

</property> <property> <name>oozie.launcher.mapreduce.map.java.opts</name> <value>-verbose</value> </property> <property> <name>mapreduce.task.classpath.user.precedence</name> <value>true</value> </property> <property> <name>oozie.launcher.mapreduce.task.classpath.user.precedence</name> <value>true</value> </property> <property> <name>oozie.launcher.mapreduce.job.user.classpath.first</name> <value>true</value> </property> </configuration> grep Loaded out | awk -F'/' '{print $NF}' | sort -u | grep jackson jackson-annotations-2.7.8.jar] jackson-core-2.2.3.jar] jackson-core-asl-1.9.13.jar] jackson-databind-2.6.5.jar] jackson-mapper-asl-1.9.13.jar] For No reason oozie java action loaded version 2.2.3 of jackson-core libraray (Even though my lib folder has a later version available)

Don't have an account?