NiFi: Flowfile retries

[#1]

For e.g. PutHDFS processor, if the failure relationship is connected to itself

1) How to control the number of retries of a single flowfile?

2) I see "FlowFile Expiration" can be used to expire the message (meaning discard the Flowfile content if it cannot be reprocessed within a specified period of time). Is there a way to retain the FlowFile content after specified number of retries, that is to persist on a different channel say local file system (PutFile) and perhaps be able to send email notification ?

[#2]

I am not aware of a setting to specify a number of retry attempts. If you think this would be a common requirement, please file a Jira requesting the feature for Apache NiFi. You can route data through a series of processors a specified number of times using a loop as demonstrated by Koji Kawamura here. You can also pair an UpdateAttribute processor with your PutHDFS so when following the failure relationship, an attribute is added/updated indicating the number of failed tries. You can then route continually-failed data to your fallback flow (PutFile & PutEmail, etc.).

Flowfile expiration refers to the age (amount of time since the flowfile entered NiFi) before the flowfile should be dropped, so it is not an ideal fit for this use case.

[#3 · Accepted solution]

@kumar

Check out this template as it will do exactly what you are looking for:

Retry_Count_Loop.xml

Just feed your failure relationship in to this process group and the output from this process group back to your processor.

Thanks,

Matt

[#4]

@Matt Clarke

hi using invoke HTTP and retry relationship is connected to itself and i want to control reties up to some count and on time that is it retry at certain time .at the same time i want to limit reties up to 5 counts..

[#5]

@umair ahmed

The Retry loop template above allows you to configure the number of retry attempts before existing the loop. I am not sure what you mean by "on time that is it retry at certain time". If the intent is to slow how fast the FlowFile is retired, you could add an additional routeOnAttribute processor to the failure loop to to loop until file has aged x amount of time.

Thanks,

Matt

[#6]

Hi umair,

@umair_ahmed , @Wynner , @Matt Clarke

The retry mechanism of the unmatched records from the invoke HTTP which was very helpful. Could you please more elaborate information on the advanced settings of the Update attribute. And the Route attribute and Execute stream command.

1) 5 times it needs to retry but only four times it has been mentioned in advanced property of the update attribute. And the expression on the advanced setting of the update attribute needs to be : ${retry.counter:replaceNull('0'):equals('1')} whether it needs to be in the incremented fashion like 

sleeponeminute -> ${retry.counter:replaceNull('0'):equals('1')}

sleepthirtyminute ->  ${retry.counter:replaceNull('0'):equals('2')}

sleeponehour ->  ${retry.counter:replaceNull('0'):equals('3')}

sleeptwohours ->  ${retry.counter:replaceNull('0'):equals('4')}

sleepmaxcount -> ${retry.counter:replaceNull('0'):equals('5')} (i.e., the sleeptime for the sleepmaxcount rule set to 180m) 

2) For the RouteOnAttribute, I have set the 5 conditions as per your screenshot. I can see the flowfile between the connector updateattribute and RouteOnAttribute then i couldn't see the flowfile in the next connector. 

Could you please elaborate more on the property setting of the other two processors Routeon Attribute and Execute StreamCommand(CommandPrompt property: It states take the environmental value ). Working Directory and Output destination attribute could you please explain these properties as well.

Thanks in Advance,

Karthick 

[#7]

@Karthick93 , As this thread was marked 'Solved' in 2016, you would have a better chance of receiving a useful response by starting a new thread. This will also provide you with the opportunity to provide details specific to your issue that could aid others in providing a more tailored answer to your question.

[#8]

@Matt Clarke

thanks for fast replying

"on time that is it retry at certain time .at the same time i want to limit reties up to 5 counts.." i mean i want do combination of count and time for limiting retries

means...that

first retry after 1 min

second after 30 min vice versa

limiting retry up to 5 count

thanks

umair

[#9]

@umair ahmed

Just spoke with Dave and he cleaned up his template/response to you here:

https://community.hortonworks.com/questions/101496/nifi-invokehttp-retries.html#answer-101588

His solution for triggering sleep based on retry count set in my template is perfect for meeting your needs. It also scales very easily by simply adding additional timer rules to the advanced UI of the UpdateAttribute processor.

Thanks,

Matt

[#10]

@Matt Clarke

In ExecuteStreamCommand processor command path= ' usr/bin/sleep' what is this thing because i am getting exception in ExecuteStreamCommand processor .."The System cannot find the file specified" what is this file

i am not getting it

and i am using windows

thanks

umair