Currently we are using oozie workflow(consists of hive,pig,sqoop actions) using below command in dev environment. In Production environment we should not run manually.can I create a shell script for below command and I can run that shell script using crontab scheduler.Is my approach is correct if yes what is the timings for the script. If not what is the approach to run below command in Production?
If you want to use crontab that you already decided to use a time-trigger/interval right? You should really use a coordinator.
If you really want to stick with crontab than the command is more or less correct. You have a typo (--oozie should be -oozie, and the port normally is 11000 but I guess you already confirmed the port?) and normally you refer to a job.properties file (stored locally not on hdfs) with -config.
@vamsi valiveti I have to side with @Santhosh B Gowda typically scheduled jobs are handled by coordinator workflow. Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action.
I will go for coordinator Job with time and data availability-based scheduling but still have following clarifications
clarification 1:- suppose if i am using below command to trigger the coordinator job.Is it one time activity in production to run this command once in production since it will trigger based on frequency for day 2?please correct me if i am wrong or do i need to run this command on day2 also?
oozie job -oozie http://sandbox.hortonworks.com:11000/oozie -config /path/to/job.properties -run
<coordinator-app name="my_first_job" start="2014-01-01T02:00Z"
clarification 2:-How to Implement condition logic in your Oozie workflow and if there's new data, run the actions, otherwise proceed to end action?