Support Questions
Find answers, ask questions, and share your expertise

NiFi: Flowfile retries

Rising Star

For e.g. PutHDFS processor, if the failure relationship is connected to itself

1) How to control the number of retries of a single flowfile?

2) I see "FlowFile Expiration" can be used to expire the message (meaning discard the Flowfile content if it cannot be reprocessed within a specified period of time). Is there a way to retain the FlowFile content after specified number of retries, that is to persist on a different channel say local file system (PutFile) and perhaps be able to send email notification ?

1 ACCEPTED SOLUTION

Master Guru

@kumar

Check out this template as it will do exactly what you are looking for:

Retry_Count_Loop.xml

Just feed your failure relationship in to this process group and the output from this process group back to your processor.

Thanks,

Matt

View solution in original post

13 REPLIES 13

I am not aware of a setting to specify a number of retry attempts. If you think this would be a common requirement, please file a Jira requesting the feature for Apache NiFi. You can route data through a series of processors a specified number of times using a loop as demonstrated by Koji Kawamura here. You can also pair an UpdateAttribute processor with your PutHDFS so when following the failure relationship, an attribute is added/updated indicating the number of failed tries. You can then route continually-failed data to your fallback flow (PutFile & PutEmail, etc.).

Flowfile expiration refers to the age (amount of time since the flowfile entered NiFi) before the flowfile should be dropped, so it is not an ideal fit for this use case.

Master Guru

@kumar

Check out this template as it will do exactly what you are looking for:

Retry_Count_Loop.xml

Just feed your failure relationship in to this process group and the output from this process group back to your processor.

Thanks,

Matt

Explorer

@Matt Clarke

hi using invoke HTTP and retry relationship is connected to itself and i want to control reties up to some count and on time that is it retry at certain time .at the same time i want to limit reties up to 5 counts..

Master Guru

@umair ahmed

The Retry loop template above allows you to configure the number of retry attempts before existing the loop. I am not sure what you mean by "on time that is it retry at certain time". If the intent is to slow how fast the FlowFile is retired, you could add an additional routeOnAttribute processor to the failure loop to to loop until file has aged x amount of time.

Thanks,

Matt

New Contributor

Hi umair,

@umair_ahmed , @Wynner , @Matt Clarke

The retry mechanism of the unmatched records from the invoke HTTP which was very helpful. Could you please more elaborate information on the advanced settings of the Update attribute. And the Route attribute and Execute stream command.

1) 5 times it needs to retry but only four times it has been mentioned in advanced property of the update attribute. And the expression on the advanced setting of the update attribute needs to be : ${retry.counter:replaceNull('0'):equals('1')} whether it needs to be in the incremented fashion like 

sleeponeminute -> ${retry.counter:replaceNull('0'):equals('1')}

sleepthirtyminute ->  ${retry.counter:replaceNull('0'):equals('2')}

sleeponehour ->  ${retry.counter:replaceNull('0'):equals('3')}

sleeptwohours ->  ${retry.counter:replaceNull('0'):equals('4')}

sleepmaxcount -> ${retry.counter:replaceNull('0'):equals('5')} (i.e., the sleeptime for the sleepmaxcount rule set to 180m) 

2) For the RouteOnAttribute, I have set the 5 conditions as per your screenshot. I can see the flowfile between the connector updateattribute and RouteOnAttribute then i couldn't see the flowfile in the next connector. 

Could you please elaborate more on the property setting of the other two processors Routeon Attribute and Execute StreamCommand(CommandPrompt property: It states take the environmental value ). Working Directory and Output destination attribute could you please explain these properties as well.

 

Thanks in Advance,

Karthick 

 

Community Manager

@Karthick93 , As this thread was marked 'Solved' in 2016, you would have a better chance of receiving a useful response by starting a new thread. This will also provide you with the opportunity to provide details specific to your issue that could aid others in providing a more tailored answer to your question.



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

Explorer

@Matt Clarke

thanks for fast replying

"on time that is it retry at certain time .at the same time i want to limit reties up to 5 counts.." i mean i want do combination of count and time for limiting retries

means...that

first retry after 1 min

second after 30 min vice versa

limiting retry up to 5 count

thanks

umair

Master Guru

@umair ahmed

Just spoke with Dave and he cleaned up his template/response to you here:

https://community.hortonworks.com/questions/101496/nifi-invokehttp-retries.html#answer-101588

His solution for triggering sleep based on retry count set in my template is perfect for meeting your needs. It also scales very easily by simply adding additional timer rules to the advanced UI of the UpdateAttribute processor.

Thanks,

Matt

Explorer

@Matt Clarke

In ExecuteStreamCommand processor command path= ' usr/bin/sleep' what is this thing because i am getting exception in ExecuteStreamCommand processor .."The System cannot find the file specified" what is this file

i am not getting it

and i am using windows

thanks

umair

Master Guru

The "sleep" command is a linux command. The command simply runs and waits the configured amount of time before exiting. Typically this command in linux is found under /usr/bin/sleep. I noticed above is missing the leading "/" . But if it is still not found, try searching your linux for it. It is installed as part of the linux "coreutils" rpm.

Matt

Explorer

@Matt Clarke

ok what is alternative command for windows .i already tried timeout and waitfor but both not work for me

Explorer

@Matt Clarke

thanks Matt for your help

umair

Master Guru

@umair ahmed

I am not a Windows Admin and could not tell you off teh top of my head.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.