Support Questions

Find answers, ask questions, and share your expertise

Nifi job scheduling

avatar

Hi,

Is there a way to kick off multiple Nifi jobs in parallel ? For example, when it comes to sqoop we can kick off 10 different sqoop jobs at the same time in oozie, because they are all separate , I can do that, there is no restriction on that, but if I do this in Nifi, how does that work? Appreciate any help.

1 ACCEPTED SOLUTION

avatar

@Pramod Kalvala You can schedule multiple jobs at a single given time assuming you have the resource available to cater all of them 🙂

Coming to how can you do that, there are multiple scheduling options available in NiFi. If you right-click on your "Triggering processor", that is the very first processor in your job and click on "Configure", you will see a scheduling tab. You will see an interface as presented below.

64686-screen-shot-2018-03-18-at-95726-am.png

There you can see "Scheduling Strategy" drop down. There you can see two major scheduling options.

  • Timer Driven
  • Cron Driven

Timer Driven strategy executes the processor according to the duration mentioned in the "Run schedule". So in this case, the processor will run every second, triggering the next processors in the flow, provided they don't have a schedule of there own.

Cron Driven is the strategy where you can mention a specific time of a day, specific day(s) etc, i.e. the schedule when you can execute that processor. Let's say that you want to run your job @1 PM every day, the Scheduling tab will look something like

64687-screen-shot-2018-03-18-at-100249-am.png

You can have whatever number of jobs scheduled to run at 1 PM by using the same scheduling strategy for all of them and all of them will run at the same time. All of those jobs will be separate will not interfere with each other, until you instruct them to do so and will run without any issues, provided that you have sufficient resources for all of them.

View solution in original post

2 REPLIES 2

avatar

@Pramod Kalvala You can schedule multiple jobs at a single given time assuming you have the resource available to cater all of them 🙂

Coming to how can you do that, there are multiple scheduling options available in NiFi. If you right-click on your "Triggering processor", that is the very first processor in your job and click on "Configure", you will see a scheduling tab. You will see an interface as presented below.

64686-screen-shot-2018-03-18-at-95726-am.png

There you can see "Scheduling Strategy" drop down. There you can see two major scheduling options.

  • Timer Driven
  • Cron Driven

Timer Driven strategy executes the processor according to the duration mentioned in the "Run schedule". So in this case, the processor will run every second, triggering the next processors in the flow, provided they don't have a schedule of there own.

Cron Driven is the strategy where you can mention a specific time of a day, specific day(s) etc, i.e. the schedule when you can execute that processor. Let's say that you want to run your job @1 PM every day, the Scheduling tab will look something like

64687-screen-shot-2018-03-18-at-100249-am.png

You can have whatever number of jobs scheduled to run at 1 PM by using the same scheduling strategy for all of them and all of them will run at the same time. All of those jobs will be separate will not interfere with each other, until you instruct them to do so and will run without any issues, provided that you have sufficient resources for all of them.

avatar

@ Rahul soni - Thanks so much for the detailed explanation.we have a custom scheduler which decides the number of jobs to run and this triggers the oozie API, which triggers the particular workflow like sqoop or Kafka whatever.So this workflow will go get the properties for the respective workflow and run the jobs.We are looking forward to replace the sqoop workflow with nifi.I have been doing some POC’s on nifi lately and realized that it has the capability to handle the load and was not sure how it would work in scheduling jobs like sqoop.Anyway your answer has given me a benefit of hope.