question Re: NiFi: Flowfile retries in Archives of Support Questions (Read Only)

NiFi: Flowfile retries

ashsskum — Thu, 10 Nov 2016 13:18:00 GMT

For e.g. PutHDFS processor, if the failure relationship is connected to itself

1) How to control the number of retries of a single flowfile?

2) I see "FlowFile Expiration" can be used to expire the message (meaning discard the Flowfile content if it cannot be reprocessed within a specified period of time). Is there a way to retain the FlowFile content after specified number of retries, that is to persist on a different channel say local file system (PutFile) and perhaps be able to send email notification ?

Re: NiFi: Flowfile retries

alopresto — Thu, 10 Nov 2016 13:44:45 GMT

I am not aware of a setting to specify a number of retry attempts. If you think this would be a common requirement, please file a Jira requesting the feature for Apache NiFi. You can route data through a series of processors a specified number of times using a loop as demonstrated by Koji Kawamura here. You can also pair an UpdateAttribute processor with your PutHDFS so when following the failure relationship, an attribute is added/updated indicating the number of failed tries. You can then route continually-failed data to your fallback flow (PutFile & PutEmail, etc.).

Flowfile expiration refers to the age (amount of time since the flowfile entered NiFi) before the flowfile should be dropped, so it is not an ideal fit for this use case.

Re: NiFi: Flowfile retries

MattWho — Thu, 10 Nov 2016 21:01:45 GMT

@kumar

Check out this template as it will do exactly what you are looking for:

Retry_Count_Loop.xml

Just feed your failure relationship in to this process group and the output from this process group back to your processor.

Thanks,

Matt

Re: NiFi: Flowfile retries

umair_ahmed — Mon, 08 May 2017 13:27:58 GMT

@Matt Clarke

hi using invoke HTTP and retry relationship is connected to itself and i want to control reties up to some count and on time that is it retry at certain time .at the same time i want to limit reties up to 5 counts..

Re: NiFi: Flowfile retries

MattWho — Mon, 08 May 2017 20:00:47 GMT

@umair ahmed

The Retry loop template above allows you to configure the number of retry attempts before existing the loop. I am not sure what you mean by "on time that is it retry at certain time". If the intent is to slow how fast the FlowFile is retired, you could add an additional routeOnAttribute processor to the failure loop to to loop until file has aged x amount of time.

Thanks,

Matt

Re: NiFi: Flowfile retries

umair_ahmed — Mon, 08 May 2017 20:23:03 GMT

@Matt Clarke

thanks for fast replying

"on time that is it retry at certain time .at the same time i want to limit reties up to 5 counts.." i mean i want do combination of count and time for limiting retries

means...that

first retry after 1 min

second after 30 min vice versa

limiting retry up to 5 count

thanks

umair

Re: NiFi: Flowfile retries

MattWho — Mon, 08 May 2017 21:13:40 GMT

@umair ahmed

Just spoke with Dave and he cleaned up his template/response to you here:

https://community.hortonworks.com/questions/101496/nifi-invokehttp-retries.html#answer-101588

His solution for triggering sleep based on retry count set in my template is perfect for meeting your needs. It also scales very easily by simply adding additional timer rules to the advanced UI of the UpdateAttribute processor.

Thanks,

Matt

Re: NiFi: Flowfile retries

umair_ahmed — Tue, 09 May 2017 13:37:24 GMT

@Matt Clarke

In ExecuteStreamCommand processor command path= ' usr/bin/sleep' what is this thing because i am getting exception in ExecuteStreamCommand processor .."The System cannot find the file specified" what is this file

i am not getting it

and i am using windows

thanks

umair

Re: NiFi: Flowfile retries

MattWho — Tue, 09 May 2017 19:41:02 GMT

The "sleep" command is a linux command. The command simply runs and waits the configured amount of time before exiting. Typically this command in linux is found under /usr/bin/sleep. I noticed above is missing the leading "/" . But if it is still not found, try searching your linux for it. It is installed as part of the linux "coreutils" rpm.

Matt

Re: NiFi: Flowfile retries

umair_ahmed — Wed, 10 May 2017 12:03:16 GMT

@Matt Clarke

ok what is alternative command for windows .i already tried timeout and waitfor but both not work for me

Re: NiFi: Flowfile retries

MattWho — Wed, 10 May 2017 19:36:11 GMT

@umair ahmed

I am not a Windows Admin and could not tell you off teh top of my head.

Re: NiFi: Flowfile retries

umair_ahmed — Wed, 10 May 2017 19:57:30 GMT

@Matt Clarke

thanks Matt for your help

umair

Re: NiFi: Flowfile retries

Karthick93 — Thu, 04 Jun 2020 07:21:41 GMT

Hi umair,

@umair_ahmed , @Wynner , @Matt Clarke

The retry mechanism of the unmatched records from the invoke HTTP which was very helpful. Could you please more elaborate information on the advanced settings of the Update attribute. And the Route attribute and Execute stream command.

1) 5 times it needs to retry but only four times it has been mentioned in advanced property of the update attribute. And the expression on the advanced setting of the update attribute needs to be : ${retry.counter:replaceNull('0'):equals('1')} whether it needs to be in the incremented fashion like

sleeponeminute -> ${retry.counter:replaceNull('0'):equals('1')}

sleepthirtyminute -> ${retry.counter:replaceNull('0'):equals('2')}

sleeponehour -> ${retry.counter:replaceNull('0'):equals('3')}

sleeptwohours -> ${retry.counter:replaceNull('0'):equals('4')}

sleepmaxcount -> ${retry.counter:replaceNull('0'):equals('5')} (i.e., the sleeptime for the sleepmaxcount rule set to 180m)

2) For the RouteOnAttribute, I have set the 5 conditions as per your screenshot. I can see the flowfile between the connector updateattribute and RouteOnAttribute then i couldn't see the flowfile in the next connector.

Could you please elaborate more on the property setting of the other two processors Routeon Attribute and Execute StreamCommand(CommandPrompt property: It states take the environmental value ). Working Directory and Output destination attribute could you please explain these properties as well.

Thanks in Advance,

Karthick

Re: NiFi: Flowfile retries

VidyaSargur — Thu, 04 Jun 2020 09:31:16 GMT

@Karthick93 , As this thread was marked 'Solved' in 2016, you would have a better chance of receiving a useful response by starting a new thread. This will also provide you with the opportunity to provide details specific to your issue that could aid others in providing a more tailored answer to your question.

Re: NiFi: Flowfile retries

Shakti — Wed, 12 Jun 2024 09:49:33 GMT

The template is not working now. Are there any alternatives??