Support Questions
Find answers, ask questions, and share your expertise

how to schedule incremental import from db2 to hive for every one hoour?

Highlighted

how to schedule incremental import from db2 to hive for every one hoour?

New Contributor

I need to schedule the job for very one hour

2 REPLIES 2

Re: how to schedule incremental import from db2 to hive for every one hoour?

Super Guru
Highlighted

Re: how to schedule incremental import from db2 to hive for every one hoour?

Super Collaborator

Do you have a working sqoop command?

With that information, you can create an hourly oozie job.

Start with a one-off workflow.xml file - find the documentation here.

https://oozie.apache.org/docs/4.2.0/DG_SqoopActionExtension.html

Make sure you can run the workflow before working on the coordinator.

Then you can make an hourly coordinator like this and put the "jobStart" and "jobEnd" properties in the oozie config file.

<coordinator-app name="DB2-Export"
  frequency="${coord:hours(1)}"
  start="${jobStart}" end="${jobEnd}" timezone="UTC"
  xmlns="uri:oozie:coordinator:0.2">
  <controls>
    <concurrency>1</concurrency>
    <execution>FIFO</execution>
    <throttle>1</throttle>
  </controls>
  <action>
    <workflow>
      <app-path>${wf_application_path}</app-path>
   </workflow>
  </action>
</coordinator-app>

You would execute this like

oozie job -config db2-export-cooord.properties -run

Where that property file might contain

jobTracker=namenode.fqdn:8050 
nameNode=hdfs://hadoop_cluster
wf_application_path=hdfs://path/to/db2-export/
oozie.coord.application.path=${wf_application_path}
jobStart=2017-11-01T09:00Z
jobEnd=2099-11-09T09:00Z