Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi works on 2 tasks instead of one

Nifi works on 2 tasks instead of one

New Contributor

92939-capture-nifi03.png

Hello everyone,

I have a problem with using nifi 'GenerateFlowFile' and 'PutLambda' processors.

My flow is very simple:

GenerateFlowFile -> PutLambda

First I want to generate exactly one flow file and than I want to invoke a lambda function exactly one time. However, for some reason, the GenerateFlowFile processor creates 2 flow files instead of one, even though it is configured to run only on the primary node with one concurrent task. This causes the lambda to be invoked 2 times.

Furthermore, I also changed the configuration of the PutLambda processor to run only on the primary node with one concurrent task, but it still invokes 2 lambdas at the same time which is causing errors.

It's important to say that this happens on a regular basis on two different clusters.

Can anyone tell me how can I fix this problem? I thought about adding a MergeContent processor after the GenerateFlowFile, but I hope you guys will have a better idea.

thanks!

92938-capture-nifi02.png


capture-nifi01.png
4 REPLIES 4

Re: Nifi works on 2 tasks instead of one

Super Guru
@gal itzhak

There is a jira regards to similar kind issue NIFI-3242,

To work around this issue use Control Rate processor after Generate Flowfile processor and configure control rate processor as

93385-screen-shot-2018-10-21-at-100757-am.png

Now processor releases 1 flowfile per minute and configure the queue before ControlRate processor to expire 10 sec.

93386-screen-shot-2018-10-21-at-100957-am.png

With this configurations we are releasing 1 flowfile per minute and another flowfile is queued and will expire in 10 sec.

Flow:

1.GenerateFlowFile
 --> success relationship(configure to expire to 10 sec) connect to controlrate processor
2.ControlRate
 --> success relationship connect to putlambda processor
3.PutLambda

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

Re: Nifi works on 2 tasks instead of one

New Contributor

thank you very much! I will try that

Re: Nifi works on 2 tasks instead of one

New Contributor

It keeps producing 2 flowfiles - the 'Control Rate' processor also works on 2 nodes, so it let's 2 flowfiles pass each minute.

I keep getting errors about primary node being changed that might be related. Have an idea how can I avoid them?

Re: Nifi works on 2 tasks instead of one

Super Guru
@gal itzhak

There is a jira related to these kind of issue also,There are several reaspns that affect primaru node being changed.

Refer to this link and change the configs accordingly, Restart the nifi instance and check if the issue has been resolved or not.

Don't have an account?
Coming from Hortonworks? Activate your account here