question Can I ensure that my own jars have classpath priority for oozie java actions? in Support Questions

Can I ensure that my own jars have classpath priority for oozie java actions?

Aaron_Dossett — Wed, 16 Dec 2015 06:01:05 GMT

I want my own jar files searched first before, for example, the standard jar files included in oozie actions like "/hadoop/yarn/local/filecache/22/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/"

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

aervits — Wed, 16 Dec 2015 21:33:43 GMT

for example you want to use pig 0.14 instead of 0.15 shipped with HDP 2.3

put 0.14 libs in sharelib with unique name, i.e. pig-14

in your job.properties specify the following

oozie.action.sharelib.for.pig=pig-14

job.properties precedence is at a job level, if you put this property within pig action, it will only take effect during the pig action and at server level, you'd put this property in oozie-site.xml

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Aaron_Dossett — Wed, 16 Dec 2015 21:50:14 GMT

Thanks. Is there a way for an individual user to do without access to add things to sharelib?

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

aervits — Wed, 16 Dec 2015 21:52:40 GMT

then you probably need to create a lib folder in your workflow directory and put your libs there. Here's a sample for a file I need to reference in my workflow. You also may not need to set configs with lib folder present. https://github.com/dbist/oozie/tree/master/apps/hcatalog

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Aaron_Dossett — Wed, 16 Dec 2015 22:00:14 GMT

I do have a lib folder, it just seems to be last in classpath precedence. A way to force it to have a higher precedence would be ideal.

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

aervits — Wed, 16 Dec 2015 22:16:48 GMT

turn it off

oozie.use.system.libpath=false

and specify your own in job.properties

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Aaron_Dossett — Thu, 17 Dec 2015 01:38:30 GMT

Thanks for all the suggestions! That did not solve the problem unfortunately. There's appears to be a lot of things still in my classpath ahead of my lib directory.

System.out.println("com.fasterxml.jackson.core.JsonFactory: [" + cl.getResource("com/fasterxml/jackson/core/JsonFactory.class") + "]");

Which prints this when run as a java action, regadless of the libpath setting

com.fasterxml.jackson.core.JsonFactory: [jar:file:/hadoop/yarn/local/filecache/22/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/jackson-core-2.2.3.jar!/com/fasterxml/jackson/core/JsonFactory.class]

Re: Can I ensure that my own jars have classpath priority for oozie java actions?

Aaron_Dossett — Thu, 17 Dec 2015 08:33:26 GMT

To answer my own question, the oozie user mailing list led me to this option within the java action configuration:

<property>
    <name>oozie.launcher.mapreduce.user.classpath.first</name>
    <value>true</value>
</property>

https://mail-archives.apache.org/mod_mbox/oozie-us...

See also OOZIE-2066: https://issues.apache.org/jira/browse/OOZIE-2066