I want to call multiple sources with same process group with different parameters in parallel in NIFI

[#1]

We have multiple different sources which we are using same process group. We need to call same process group with different parameters in parallel.

[#2]

@ShyamKumar, if you expect an answer, try explaining exactly what you are trying to achieve, what you tried and did not work - why it did not work and so on 🙂 In addition, try saying the version of NiFi you are using and provide some further details. 

Based on how you wrote the post, you can use expect a generic answer like you can use a Remote Process Group to call a specific Process Group from your NiFi Instance. Another option would be to use the combination of Input Port and Output Port. Or you can even develop a custom script and execute it on a scripted processor. 

Again, to get a tailored answer to your use case, try providing some more details 🙂 

[#3]

Version - 1.20.0

We have different client which will call the same generic PG. If multiple clients are calling the same generic pg in parallel how we can seperate the PGs without any manual intervention. We have scheduling here through airflow.

[#4]

@ShyamKumar 

Your dataflow design is still unclear here.
- 'We have different client which will call the same generic PG" --> How is this being done?  When you say "clients", are you referring to external to NiFi clients?  How are these client request being sent to/received by NiFi's PG?   

We would need to understand your dataflow better before being able to provide better feedback.  A detailed use case would be very helpful here.

Thanks,
Matt

[#5]

@ShyamKumar 

A NiFi Process Group (PG) can only have one parameter context assigned to it.  You can allow the PG configured parameter context to inherit parameters from other parameter context(s). Only parameter which are not explicitly defined in the PG configured parameter context will be inherited.  You can not have multiple parameters of same name defined.  Changing the values assigned to parameter requires the stopping and starting of components configured to use that parameter.

So it is not possible to pass FlowFiles to the same PG and specify using unique contexts per FlowFile.   You would need to copy the PG and assign each copy a different parameter context for the properties that are unique and inherit all the parameters that are shared by all FlowFiles from some parent parameter context.

While all component properties support being parametrized, a good number of component properties support NiFi Expression Language (NEL).  If the properties that need to be unique per FlowFile all support NEL, you could set those attributes on the FlowFiles upstream of PG and those attributes could be used uniquely tailor the execution of processor components per FlowFile.  

As far as parallel execution goes, that is controlled via the concurrent tasks setting on a processor component.  By default the value is 1 meaning only a single execution at time.  but setting this to a higher value would allow parallel execution against different FlowFiles.  Be very cautious with adjusting this value and monitor your servers CPU load average.  You want to use the smallest value possible to achieve the highest throughput achievable on your hardware running your dataflow(s).  Setting to high could just lead to this processor consuming more time then needed on the CPU and preventing other processors from efficiently being able to use CPU.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt