Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can I ensure that my own jars have classpath priority for oozie java actions?

avatar
Rising Star

I want my own jar files searched first before, for example, the standard jar files included in oozie actions like "/hadoop/yarn/local/filecache/22/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/"

1 ACCEPTED SOLUTION

avatar
Rising Star

To answer my own question, the oozie user mailing list led me to this option within the java action configuration:

<property>
    <name>oozie.launcher.mapreduce.user.classpath.first</name>
    <value>true</value>
</property>

https://mail-archives.apache.org/mod_mbox/oozie-us...

See also OOZIE-2066: https://issues.apache.org/jira/browse/OOZIE-2066

View solution in original post

7 REPLIES 7

avatar
Master Mentor

for example you want to use pig 0.14 instead of 0.15 shipped with HDP 2.3

put 0.14 libs in sharelib with unique name, i.e. pig-14

in your job.properties specify the following

oozie.action.sharelib.for.pig=pig-14

job.properties precedence is at a job level, if you put this property within pig action, it will only take effect during the pig action and at server level, you'd put this property in oozie-site.xml

avatar
Rising Star

Thanks. Is there a way for an individual user to do without access to add things to sharelib?

avatar
Master Mentor

then you probably need to create a lib folder in your workflow directory and put your libs there. Here's a sample for a file I need to reference in my workflow. You also may not need to set configs with lib folder present. https://github.com/dbist/oozie/tree/master/apps/hcatalog

avatar
Rising Star

I do have a lib folder, it just seems to be last in classpath precedence. A way to force it to have a higher precedence would be ideal.

avatar
Master Mentor

turn it off

oozie.use.system.libpath=false

and specify your own in job.properties

avatar
Rising Star

Thanks for all the suggestions! That did not solve the problem unfortunately. There's appears to be a lot of things still in my classpath ahead of my lib directory.

System.out.println("com.fasterxml.jackson.core.JsonFactory: [" + cl.getResource("com/fasterxml/jackson/core/JsonFactory.class") + "]");

Which prints this when run as a java action, regadless of the libpath setting

com.fasterxml.jackson.core.JsonFactory: [jar:file:/hadoop/yarn/local/filecache/22/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/jackson-core-2.2.3.jar!/com/fasterxml/jackson/core/JsonFactory.class]

avatar
Rising Star

To answer my own question, the oozie user mailing list led me to this option within the java action configuration:

<property>
    <name>oozie.launcher.mapreduce.user.classpath.first</name>
    <value>true</value>
</property>

https://mail-archives.apache.org/mod_mbox/oozie-us...

See also OOZIE-2066: https://issues.apache.org/jira/browse/OOZIE-2066