Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

oozie workflow in Production

avatar
Expert Contributor

Currently we are using oozie workflow(consists of hive,pig,sqoop actions) using below command in dev environment. In Production environment we should not run manually.can I create a shell script for below command and I can run that shell script using crontab scheduler.Is my approach is correct if yes what is the timings for the script. If not what is the approach to run below command in Production?

oozie job --oozie http://host_nameofoozieserver:8080/oozie -D
oozie.wf.application.path=hdfs://namenodepath/pathof_workflow_xml/workflow.xml-run 

11349-oozie-clarification.jpg

1 ACCEPTED SOLUTION

avatar
8 REPLIES 8

avatar
@vamsi valiveti

Have we explored oozie co-ordinator ? does that not solve the problem

https://oozie.apache.org/docs/3.1.3-incubating/CoordinatorFunctionalSpec.html

avatar
Expert Contributor

HI @Santhosh B Gowda

Thanks for input.

a)My question is how to run below command in production since we should not run manually.

oozie job --oozie http://host_nameofoozieserver:8080/oozie -Doozie.wf.application.path=hdfs://namenodepath/pathof_workflow_xml/workflow.xml-run 

b)I know about coordinator but at this point of time i am not sure whether i have to use data or time triggers.

currently we are running flume continusoly

avatar
Rising Star

If you want to use crontab that you already decided to use a time-trigger/interval right? You should really use a coordinator.

If you really want to stick with crontab than the command is more or less correct. You have a typo (--oozie should be -oozie, and the port normally is 11000 but I guess you already confirmed the port?) and normally you refer to a job.properties file (stored locally not on hdfs) with -config.

So it should look like:

oozie job -oozie http://sandbox.hortonworks.com:11000/oozie -config /path/to/job.properties -run

In the job.properties file you would have parameters listed like namenode, jobtracker, hcmetastoreuri and of course the one you provide via the -D: oozie.wf.application.path

Normally the hdfs://namenode part can be omitted from the apppath url.

avatar
Expert Contributor

Hi experts,

Any input on my clarification

avatar
Master Mentor

@vamsi valiveti I have to side with @Santhosh B Gowda typically scheduled jobs are handled by coordinator workflow. Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action.

avatar
Expert Contributor

a)Thanks for input @Artem Ervits .Your input is always appreciated.

I will go for coordinator Job with time and data availability-based scheduling but still have following clarifications

clarification 1:- suppose if i am using below command to trigger the coordinator job.Is it one time activity in production to run this command once in production since it will trigger based on frequency for day 2?please correct me if i am wrong or do i need to run this command on day2 also?

oozie job -oozie http://sandbox.hortonworks.com:11000/oozie -config /path/to/job.properties -run


<coordinator-app name="my_first_job" start="2014-01-01T02:00Z"
end="2014-12-31T02:00Z" frequency="${coord:days(1)}"
xmlns="uri:oozie:coordinator:0.4">
clarification 2:-How to Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action?

avatar
Expert Contributor

Hi folks

Any input on my clarification

avatar
Master Mentor

1. Only once

2. Use decision property https://www.infoq.com/articles/oozieexample/