Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Trigerring a processor making use of the event driven scheduling strategy

Highlighted

Trigerring a processor making use of the event driven scheduling strategy

I wanted to know if there is a way to trigger the start of a processor in NiFi only if two of the incoming connections send FlowFiles to the processor. Now, when I tested the event driven scheduling strategy, the processor gets triggered by any flowfile that flows into the processor first. For this use case, I would like to trigger the start of a processor only if certain processors finish their execution and forward their flowfiles and only after all of the files from these processors are received, it initiates the following processor. But, the predecessor processors finish execution at different time intervals and even if I use routeonAttribute, both of these different connections don't flow through that at the same time and thus the AND condition is not fulfilled. Any other way to trigger a processor only if multiple predecessor processors finish execution?

3 REPLIES 3
Highlighted

Re: Trigerring a processor making use of the event driven scheduling strategy

You can try to use "MergeContent" processor before RouteOnAttribute. In merge content you can try to merge flowfiles coming from different upstream processors (though not recommanded). You may need to use "Funnel" to get flowfiles into single connection. For e.g. if u have 4 upstream processors then u can use Funnel and then MergeContent processor to merge 4 flow files and downstream RoutOnAttribute processor will execute only after it receives merged flow file.

However this may misfire if more than one flowfile is received from same processor.e.g. if u have processors A,B,C and D.If you receive flowfiles A,B,C and D and are merged then it will work fine. However if you receive 2 flowfiles from A and 1 from B and C, then 4 flowfiles are merged and it misfires downstream RouteOnAttribute processor though flowfile from D is not received. So you need to be careful about scheduling interval.

Highlighted

Re: Trigerring a processor making use of the event driven scheduling strategy

This is exactly the approach I took and I have the same worry. Even though routeOnAttribute will only fire if it receives all the attributes from the different upstream processors as it won't match the condition, it is still a problem if the merge content processes and sends the merged flow file forward without waiting for the other incoming connection to send a flow file. It won't be a solution that can be taken to a production environment. That is the reason I had asked if the event driven scheduling strategy can be modified to wait for multiple incoming connections to trigger the start of a processor.

Highlighted

Re: Trigerring a processor making use of the event driven scheduling strategy

I think backpressure can configured to 1 flowfile each connection from upstream processor. That will ensure that only one flowfile is received from each processor before merging content.

Don't have an account?
Coming from Hortonworks? Activate your account here