Reply
Contributor
Posts: 50
Registered: ‎08-05-2015
Accepted Solution

Oozie job.properties - Local or HDFS?

[ Edited ]

When I'm using Oozie, I tend to use Hue to submit coordinators and workflows because it's simple.  As a result, I keep my job.properties files on HDFS, in the same folder of its aligned workflow.xml or coordinator.xml.  I read in books and other reference material that the job.properties file should be on the local file system.  This makes sense if I issue through the command-line like so:

oozie   job -run -config  job.properties 

 

So, knowing I and others prefer interfacing through Hue for simplicity, which is the better practice, or is it completely preferential? (note that my workflow.xml and coordinator.xml files are on HDFS no matter what)

 

(Option 1) I ignore the reference material in favor of keeping the job.properties in HDFS so that I can continue submitting and managing through Hue

(Option 2) Keep the job.properties file on the local file system and issue through the command-line.

 

Appreciate the dialog!

Posts: 1,760
Kudos: 379
Solutions: 282
Registered: ‎07-31-2013

Re: Oozie job.properties - Local or HDFS

The point about having the job.properties on the local filesystem is just
within the context of using Oozie CLI - it is explicitly mentioned to avoid
confusion in the process of writing a workflow XML (the XML needs to be on
HDFS, but the properties are a local file read by the Oozie CLI when
invoked - people often get confused by this relationship).

When you use Hue, which in turn simply uses the Oozie REST API, you can use
whatever mechanism Hue offers you to manage your job.properties (as a HDFS
file, defined within-workflow, etc.).

The properties are something used to resolve the workflow's variables and
supply it some necessary parameters. Once submitted, the properties file or
list of properties do not matter anymore to Oozie.
Contributor
Posts: 50
Registered: ‎08-05-2015

Re: Oozie job.properties - Local or HDFS

Thank you for the explanation Harsh J.  You hit on points that I'm sure others will find valuable along the way as well. 

 

From your response, I gather I have options and it comes down to preference, which is what I was hoping for. :)

Explorer
Posts: 10
Registered: ‎04-05-2017

Re: Oozie job.properties - Local or HDFS

Your answer is very confusing.  

 

Is there a single example of using a job.properties file to provide the --username and --password information needed by a jdbc:sqlserver:..... driver?  

 

Do I need to add Arguments?  I can add arguments for the entire Sqoop action command within my Hue..Oozie workflow editor and it works with HARDCODED username and password values but I don't want hardcoded values.

 

How about a screenshot showing the Sqoop Command arguments and a separate screenshot showing the Properties page?  Does anyone at Cloudera have one of these that they can share with the community?  It would make the trial and error, brute force method of trying all possible combinations of everything much easier.

 

Thanks.

AVR
New Contributor
Posts: 2
Registered: ‎07-19-2018

Re: Oozie job.properties - Local or HDFS

Hi Harsh J,

 

I'm trying to update job.properties file for my coordinator after few changes in job.properties which is in HDFS. How do I make my new job.properties file is read by coordinator when it is execute next time? 

 

Do I need to do anyting at CLI?

 

Condition: My job.properties file should be in HDFS location where workflow.xml & coordinator.xml files present.

Thanks
Venkat

 

 

Contributor
Posts: 50
Registered: ‎08-05-2015

Re: Oozie job.properties - Local or HDFS

From what I recall, the job.properties is read in only on initial submission, so any subsequent changes will not take effect.  I believe killing and re-submitting the job is what you'll need to do.  This is in contrast to changing something like a workflow.xml, which is read in anew during every execution.

Contributor
Posts: 50
Registered: ‎08-05-2015

Re: Oozie job.properties - Local or HDFS

You would probably be better served by asking this question in a separate thread.
AVR
New Contributor
Posts: 2
Registered: ‎07-19-2018

Re: Oozie job.properties - Local or HDFS

Thanks tseader. If we kill or resubmit the coordinator, we will lose history/lineage of that job.Correct?
Is there any idea how to update job.properties without kill or re-submission?
Thanks
Venkat
Announcements