NiFi - Scheduling - parts of the processors within a process group should run only once

[#1]

Hi, I am new to NiFi (using version 1.6.0) and hope someone can answer my question concerning scheduling.

Here is my process group.

When the process group gets activated the blue outlined processors only should be executed once.

Question: Is this possible? I don’t finde a possibility to get this work.

Any help is very appreciated! Thanks.

[#2]

@Ingrid Justen

In your flow ExecuteSql is the trigger processor if you want to run the processor only once then

Schedule ExecuteSQL processor as

Now the processor Run Schedule is 1 day so this will run as soon as you start for the first time then after 1 day this processor triggers again and all the other processor except of ExecuteSQL(trigger) you can schedule them 0 sec(default) so that when there some data then processor are going to triggered to process them.

Refer to this, this and this links for more details regards to Scheduling strategies of NiFi processors.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

[#3]

@Ingrid Justen

1.after 1 day when the ExecuteSQL is still active and will be triggered again. I will have to test it.

Yes processor will trigger after 1 day then it will execute the sql statement again.

2.The option "Execution" (value "Primary node") does not appear in my NiFi. Is this a version thing (working with 1.6.0)?

-> Stop ExecuteSQL processor

-> Right click on the processor
-> Goto Configure
-> Click on Scheduling tab
-> Select PrimaryNode in Execution dropdown.(this property only applicable if you are having more than one node for NiFi instance).

1.PrimaryNode //processor is scheduled to run in only primary node
2.AllNodes //if you are having 3 node nifi cluster then the processor will scheduled to run all the nodes i.e. instead of one output flowfile we are going to have 3 flowfiles with same size(this is just all nodes are doing same work results data duplication)

this and this links explains more regards to the modes of Executions in NiFi

-

For this case keep the blue outlined processors into one processor group and use NiFi RestAPI to start the processor group.

Once the processing is completed then By using RestAPI stop the processor group.

Following this way we are going to trigger ExecuteSQL only once, then we are stopping the group as soon as all the processing is done.

Refer to this, this,this links regards to stop/start processor group using NiFi RestAPI.

Refer to this link regards to stop processor group once the execution is completed.

[#4]

1. Executing the statement after 1 day again - doesn’t sound good. So I will try your suggestion putting the blue outlined processors into one process group. Stopping by using RestAPI sounds like a plan. I will have a look to the links and give it a try. As far as I have seen this RestAPI-call can then be made by the begin of the second process group with an InvokeHTTP-processor. Am I running into the right direction?

2. No, the option „Execution“ doesn’t appear. I don’t know anything about the installation I’m working on but probably it depends on it. Thanks for the links, I will try to learn more about.

I really appreciate your help and the lot of links you mentioned, great!

[#5]

Hi All,

I am doing a similar POC.

We are trying to implement batch ingestion using NiFi.

Problem statement:

I am preparing a SQL query and put the query in the processor using REST API.

But, I want to stop the processor group after the ingestion is done because for the next run I will put the query with update CDC column value.

Can someone please help me with the logic of stopping the processor after the ingestion is done, because to modify the processor, it has to be stopped

[#6]

If I understood your question correctly, you want your processor to run only once and not want to schedule it?

If that's so, You can use the CRON driven option in the Scheduling strategy to have the flow run once. Or use the rest api to run the processor once.

Here is an example using the cron option, the below processor will run once on May 8th, 2017 at 10AM

[#7]

Thanks for both of your answers!

As mentioned I have no experience with NiFi and my idea was to start the process group once for as long as there is no need to change something (hope this will be a very long time, weeks, months). If the process group has to be stopped for changes the new start should again run the blue outlined processors (the trigger ExecuteSQL) only once.

@Jatin Sablok - So I am afraid your suggestion won't do this.

@Shu - I think your hint does the trick. But I don't unterstand what happens after 1 day when the ExecuteSQL is still active and will be triggered again. I will have to test it.

One question concerning your screenshots. The option "Execution" (value "Primary node") does not appear in my NiFi. Is this a version thing (working with 1.6.0)?