nifi handling failures in failure realtionship based on message

[#1]

Hi All,

Thanks a lot to this awesome community.

Is there any way to check for failure messages in failure queue, and route it to appropriate processor back in the flow based on the failure message.

For example, if the message in the failure is "Message is too long", I want it to route it back to splittext processors and split the message and route it again.

Any suggestions or ideas

Thanks

Dheeru

[#2 · Accepted solution]

@dhieru singh

Failure relation messages won't have any write attribute(failure attribute) that is added to the flow files when the ff are transferred to failure queue.

But you can use either of below methods to check the length or size of the flow files and route to SplitText processors.

1.Checking size of the flow file and routing to split text processor.

2.Check the length of the message and route to split text processor

3.Using Route On Content processor

1.Checking size of the flow file and routing to split text processor:-

If you know the size of the flow file that will be having Message is too long then you can use Route on Attribute processor and add new property

size more than 1 byte as

${fileSize:gt(1)} //Here we are checking the flow file size is greater than 1 byte

So like this way you can filter out the flow files based on the size of the flow file but first you need to know the size of the flow files which are routing to failure.

Configs:-

Flow:-

Failure relation --> RouteonAttribute(check filesize) -->SplitText processor-->

2.Check the length of the message and route to split text processor:-

In this method we are using the same failure relation to Extract Text Processor.

Add new property in Extract Text Processor to extract all the content of the flow file to an attribute

Content as

(.*)

Configs:-

If you are following this method then you need to change the highlighted properties in above screenshot according to your flow file size.

As we are capturing everything by using (.*) regex so you need to change the

1.Max capture group length is Specifies the maximum number of characters a given capture group value can have. Any characters beyond the max will be truncated.
2.Maximum Buffer Size is Specifies the maximum amount of data to buffer (per file) in order to apply the regular expressions. Files larger than the specified maximum will not be fully evaluated.

So once we are done with this step all the contents of ff is now added as an content attribute to the ff.

RouteonAttribute Processor:-

Use RouteonAttribute processor and check for length of the content attribute by using NiFi expression language.

length more than 1 as

${content:length():gt(1)} //we are having content attribute from extract text processor and using that attribute and checking length of attribute and checking is it greater than 1.

Configs:-

Example:- if your content attribute having value as hi hello then length will be 8.

Flow:-

Failure Relation --> Extract Text(add property to capture all contents and change buffer size and capture length as per your flow file size) --> RouteOnAttribute(Check the length of attribute and route)-->SplitText processor

3.Using Route On Content processor:-

You can use RouteonContent processor to check the content of flow file by changing the properties

1.Match Requirement property change to

content must contain match

2.Buffer size depends on your flow file size

3. add property more than 1 length

[a-zA-Z0-9\s+]{1} //check content of flow file with space and Matches exactly 1 times including spaces

Configs:-

If you want to route the messages having more than 1000 length then change the regex to

[a-zA-Z0-9\s+]{1000} //Checks flow file content, matches if ff having length of messages more than 1000 including spaces matches with that message.

Flow:-

Failure relation --> RouteonContent -->SplitText

These are the ways we can filter out the failure relation messages, As you can choose which method best fit for your Case.