Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can I ensure that my own jars have classpath priority for oozie java actions?

Solved Go to solution
Highlighted

Can I ensure that my own jars have classpath priority for oozie java actions?

Contributor

I want my own jar files searched first before, for example, the standard jar files included in oozie actions like "/hadoop/yarn/local/filecache/22/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/"

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Contributor

To answer my own question, the oozie user mailing list led me to this option within the java action configuration:

<property>
    <name>oozie.launcher.mapreduce.user.classpath.first</name>
    <value>true</value>
</property>

https://mail-archives.apache.org/mod_mbox/oozie-us...

See also OOZIE-2066: https://issues.apache.org/jira/browse/OOZIE-2066

7 REPLIES 7

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Mentor

for example you want to use pig 0.14 instead of 0.15 shipped with HDP 2.3

put 0.14 libs in sharelib with unique name, i.e. pig-14

in your job.properties specify the following

oozie.action.sharelib.for.pig=pig-14

job.properties precedence is at a job level, if you put this property within pig action, it will only take effect during the pig action and at server level, you'd put this property in oozie-site.xml

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Contributor

Thanks. Is there a way for an individual user to do without access to add things to sharelib?

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Mentor

then you probably need to create a lib folder in your workflow directory and put your libs there. Here's a sample for a file I need to reference in my workflow. You also may not need to set configs with lib folder present. https://github.com/dbist/oozie/tree/master/apps/hcatalog

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Contributor

I do have a lib folder, it just seems to be last in classpath precedence. A way to force it to have a higher precedence would be ideal.

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Mentor

turn it off

oozie.use.system.libpath=false

and specify your own in job.properties

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Contributor

Thanks for all the suggestions! That did not solve the problem unfortunately. There's appears to be a lot of things still in my classpath ahead of my lib directory.

System.out.println("com.fasterxml.jackson.core.JsonFactory: [" + cl.getResource("com/fasterxml/jackson/core/JsonFactory.class") + "]");

Which prints this when run as a java action, regadless of the libpath setting

com.fasterxml.jackson.core.JsonFactory: [jar:file:/hadoop/yarn/local/filecache/22/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/jackson-core-2.2.3.jar!/com/fasterxml/jackson/core/JsonFactory.class]

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Contributor

To answer my own question, the oozie user mailing list led me to this option within the java action configuration:

<property>
    <name>oozie.launcher.mapreduce.user.classpath.first</name>
    <value>true</value>
</property>

https://mail-archives.apache.org/mod_mbox/oozie-us...

See also OOZIE-2066: https://issues.apache.org/jira/browse/OOZIE-2066