oozie workflow in Production

[#1]

Currently we are using oozie workflow(consists of hive,pig,sqoop actions) using below command in dev environment. In Production environment we should not run manually.can I create a shell script for below command and I can run that shell script using crontab scheduler.Is my approach is correct if yes what is the timings for the script. If not what is the approach to run below command in Production?

oozie job --oozie http://host_nameofoozieserver:8080/oozie -D
oozie.wf.application.path=hdfs://namenodepath/pathof_workflow_xml/workflow.xml-run 

[#2 · Accepted solution]

@vamsi valiveti

Have we explored oozie co-ordinator ? does that not solve the problem

https://oozie.apache.org/docs/3.1.3-incubating/CoordinatorFunctionalSpec.html

[#3]

HI @Santhosh B Gowda

Thanks for input.

a)My question is how to run below command in production since we should not run manually.

oozie job --oozie http://host_nameofoozieserver:8080/oozie -Doozie.wf.application.path=hdfs://namenodepath/pathof_workflow_xml/workflow.xml-run 

b)I know about coordinator but at this point of time i am not sure whether i have to use data or time triggers.

currently we are running flume continusoly

[#4]

If you want to use crontab that you already decided to use a time-trigger/interval right? You should really use a coordinator.

If you really want to stick with crontab than the command is more or less correct. You have a typo (--oozie should be -oozie, and the port normally is 11000 but I guess you already confirmed the port?) and normally you refer to a job.properties file (stored locally not on hdfs) with -config.

So it should look like:

oozie job -oozie http://sandbox.hortonworks.com:11000/oozie -config /path/to/job.properties -run

In the job.properties file you would have parameters listed like namenode, jobtracker, hcmetastoreuri and of course the one you provide via the -D: oozie.wf.application.path

Normally the hdfs://namenode part can be omitted from the apppath url.

[#5]

Hi experts,

Any input on my clarification

[#6]

@vamsi valiveti I have to side with @Santhosh B Gowda typically scheduled jobs are handled by coordinator workflow. Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action.

[#7]

a)Thanks for input @Artem Ervits .Your input is always appreciated.

I will go for coordinator Job with time and data availability-based scheduling but still have following clarifications

clarification 1:- suppose if i am using below command to trigger the coordinator job.Is it one time activity in production to run this command once in production since it will trigger based on frequency for day 2?please correct me if i am wrong or do i need to run this command on day2 also?

oozie job -oozie http://sandbox.hortonworks.com:11000/oozie -config /path/to/job.properties -run


<coordinator-app name="my_first_job" start="2014-01-01T02:00Z"
end="2014-12-31T02:00Z" frequency="${coord:days(1)}"
xmlns="uri:oozie:coordinator:0.4">

clarification 2:-How to Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action?

[#8]

Hi folks

Any input on my clarification

[#9]

1. Only once

2. Use decision property https://www.infoq.com/articles/oozieexample/