Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to setup oozie custom share lib for an action in places other then sharelib_timestamp and reference them with a name so then later we can use this in the configuration ?

how to setup oozie custom share lib for an action in places other then sharelib_timestamp and reference them with a name so then later we can use this in the configuration ?

Expert Contributor

how to setup oozie custom share lib for an action in places other then sharelib_timestamp and reference them with a name so then later we can use this in the configuration ? We are trying to stage a shared lib directory outside of the normal /users/oozie/share/libs/XXXXXXXXX

Assumption is that oozie.service.ShareLibService.mapping.file is what will acccomplish this for us

e.g. I already have

/user/oozie/share/lib/lib_$epochtime/spark

But I want to create another directory for specific spark version and call use this as share lib for my spark action.

/user/oozie/share/lib/spark_161

5 REPLIES 5
Highlighted

Re: how to setup oozie custom share lib for an action in places other then sharelib_timestamp and reference them with a name so then later we can use this in the configuration ?

Super Guru
@Saumil Mayani

You can put your custom jars in /user/oozie/share/lib/spark_161 and add below line in your job.properties

to reference them

oozie.libpath=/user/oozie/share/lib/spark_161,another/location/if/you/want
Highlighted

Re: how to setup oozie custom share lib for an action in places other then sharelib_timestamp and reference them with a name so then later we can use this in the configuration ?

Expert Contributor

Hi @Kuldeep Kulkarni can this "oozie.libpath" be used in conjunction to oozie.use.system.libpath=true. Also, how can infrastructure team enforce this for a particular action without tenants needing to change their job.properties. How does "oozie.service.ShareLibService.mapping.file" work? can this be used instead?

Highlighted

Re: how to setup oozie custom share lib for an action in places other then sharelib_timestamp and reference them with a name so then later we can use this in the configuration ?

Super Guru

@Saumil Mayani

1. Yes you can use both in job.properties.

2. If you want it to be workflow specific then there is an alternative way

- No need to add anything in job.properties or sharelib. You can just create a directory named “lib” next to your workflow.xml in HDFS and put jars in there.

3. Regarding oozie.service.ShareLibService.mapping.file --> This needs to be configured in oozie-site.xml. need service restart, I would suggest go with option 1 OR 2

Hope this information helps.

Please accept the answer if you got required information :)

Highlighted

Re: how to setup oozie custom share lib for an action in places other then sharelib_timestamp and reference them with a name so then later we can use this in the configuration ?

Expert Contributor

@Kuldeep Kulkarni I understand these options. maybe I have not added my requirement explicitly. Oozie server uses following sharlib by default for respective Action Name. /user/oozie/share/lib/lib_$epochtime/$ActionName

where $epochtime -> latest timestamp upon reboot. How do we change this default location from Oozie server side to be more deterministic manner, ($epochtime is variable). Ok to restart Oozie server when there is a change, seamless to the user submitting the job. (dont want user to update their job.properties or workflow directories)

Highlighted

Re: how to setup oozie custom share lib for an action in places other then sharelib_timestamp and reference them with a name so then later we can use this in the configuration ?

Mentor

@cnormile please advise

Don't have an account?
Coming from Hortonworks? Activate your account here