question Re: oozie workflow in Production in Archives of Support Questions (Read Only)

oozie workflow in Production

vamsi123 — Mon, 19 Aug 2019 10:07:23 GMT

Currently we are using oozie workflow(consists of hive,pig,sqoop actions) using below command in dev environment. In Production environment we should not run manually.can I create a shell script for below command and I can run that shell script using crontab scheduler.Is my approach is correct if yes what is the timings for the script. If not what is the approach to run below command in Production?

oozie job --oozie http://host_nameofoozieserver:8080/oozie -D
oozie.wf.application.path=hdfs://namenodepath/pathof_workflow_xml/workflow.xml-run

Re: oozie workflow in Production

sgowda — Thu, 12 Jan 2017 17:13:44 GMT

@vamsi valiveti

Have we explored oozie co-ordinator ? does that not solve the problem

https://oozie.apache.org/docs/3.1.3-incubating/CoordinatorFunctionalSpec.html

Re: oozie workflow in Production

vamsi123 — Thu, 12 Jan 2017 17:43:00 GMT

HI @Santhosh B Gowda

Thanks for input.

a)My question is how to run below command in production since we should not run manually.

oozie job --oozie http://host_nameofoozieserver:8080/oozie -Doozie.wf.application.path=hdfs://namenodepath/pathof_workflow_xml/workflow.xml-run

b)I know about coordinator but at this point of time i am not sure whether i have to use data or time triggers.

currently we are running flume continusoly

Re: oozie workflow in Production

vamsi123 — Thu, 12 Jan 2017 23:40:27 GMT

Hi experts,

Any input on my clarification

Re: oozie workflow in Production

aervits — Fri, 13 Jan 2017 00:25:03 GMT

@vamsi valiveti I have to side with @Santhosh B Gowda typically scheduled jobs are handled by coordinator workflow. Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action.

Re: oozie workflow in Production

rene_sluiter — Fri, 13 Jan 2017 16:38:03 GMT

If you want to use crontab that you already decided to use a time-trigger/interval right? You should really use a coordinator.

If you really want to stick with crontab than the command is more or less correct. You have a typo (--oozie should be -oozie, and the port normally is 11000 but I guess you already confirmed the port?) and normally you refer to a job.properties file (stored locally not on hdfs) with -config.

So it should look like:

oozie job -oozie http://sandbox.hortonworks.com:11000/oozie -config /path/to/job.properties -run

In the job.properties file you would have parameters listed like namenode, jobtracker, hcmetastoreuri and of course the one you provide via the -D: oozie.wf.application.path

Normally the hdfs://namenode part can be omitted from the apppath url.

Re: oozie workflow in Production

vamsi123 — Fri, 13 Jan 2017 23:51:31 GMT

a)Thanks for input @Artem Ervits .Your input is always appreciated.

I will go for coordinator Job with time and data availability-based scheduling but still have following clarifications

clarification 1:- suppose if i am using below command to trigger the coordinator job.Is it one time activity in production to run this command once in production since it will trigger based on frequency for day 2?please correct me if i am wrong or do i need to run this command on day2 also?

oozie job -oozie http://sandbox.hortonworks.com:11000/oozie -config /path/to/job.properties -run


<coordinator-app name="my_first_job" start="2014-01-01T02:00Z"
end="2014-12-31T02:00Z" frequency="${coord:days(1)}"
xmlns="uri:oozie:coordinator:0.4">

clarification 2:-How to Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action?

Re: oozie workflow in Production

vamsi123 — Tue, 17 Jan 2017 22:53:06 GMT

Hi folks

Any input on my clarification

Re: oozie workflow in Production

aervits — Tue, 17 Jan 2017 22:58:48 GMT

1. Only once

2. Use decision property https://www.infoq.com/articles/oozieexample/