Nifi works on 2 tasks instead of one

[#1]

Hello everyone,

I have a problem with using nifi 'GenerateFlowFile' and 'PutLambda' processors.

My flow is very simple:

GenerateFlowFile -> PutLambda

First I want to generate exactly one flow file and than I want to invoke a lambda function exactly one time. However, for some reason, the GenerateFlowFile processor creates 2 flow files instead of one, even though it is configured to run only on the primary node with one concurrent task. This causes the lambda to be invoked 2 times.

Furthermore, I also changed the configuration of the PutLambda processor to run only on the primary node with one concurrent task, but it still invokes 2 lambdas at the same time which is causing errors.

It's important to say that this happens on a regular basis on two different clusters.

Can anyone tell me how can I fix this problem? I thought about adding a MergeContent processor after the GenerateFlowFile, but I hope you guys will have a better idea.

thanks!

[#2]

@gal itzhak

There is a jira regards to similar kind issue NIFI-3242,

To work around this issue use Control Rate processor after Generate Flowfile processor and configure control rate processor as

Now processor releases 1 flowfile per minute and configure the queue before ControlRate processor to expire 10 sec.

With this configurations we are releasing 1 flowfile per minute and another flowfile is queued and will expire in 10 sec.

Flow:

1.GenerateFlowFile
 --> success relationship(configure to expire to 10 sec) connect to controlrate processor
2.ControlRate
 --> success relationship connect to putlambda processor
3.PutLambda

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

[#3]

thank you very much! I will try that

[#4]

It keeps producing 2 flowfiles - the 'Control Rate' processor also works on 2 nodes, so it let's 2 flowfiles pass each minute.

I keep getting errors about primary node being changed that might be related. Have an idea how can I avoid them?

[#5]

@gal itzhak

There is a jira related to these kind of issue also,There are several reaspns that affect primaru node being changed.

Refer to this link and change the configs accordingly, Restart the nifi instance and check if the issue has been resolved or not.