How to re route flow files to 4 Different processors ?

[#1]

Hi All,

please help me out to solve this issue.

Im having 10 Different flow files in the following size.

3 files size (100-500MB)

2 Files (1GB)

2 Files(1.8GB)

3 files (50MB)

I need to write some logic that should reroute these 10 files to 4 Convert Record processors running parallely ? is there any way that we could do this ?

[#2 · Accepted solution]

@Manikandan Jeyabal
You can use RouteonAttribute with fileSize attribute and check if the size of flowfile is greater than 1GB.

Example:-

1.filesize > 1gig

 ${fileSize:gt(1048576000)}

in the above routeonattribute processor i have added new property which can detect and transfer flowfiles size if the size is greater than 1GB. we need to keep the size of file in bytes.

2.filesize >100MB and <500MB

${fileSize:gt(1048576):and(${fileSize:lt(524288000) 

second property will route the flowfiles which are having sizes greater than 100MB and less than 500 MB.

Like this way you can add new properties in Routeonattribute and transfer the flowfiles to corresponding convert record processors based on Filesizes.

**1 megabyte (1048576 bytes)**

**1 Gigabyte = 1000 Megabytes**

As i'm using gt,lt(greaterthan,lessthan) based on your requirements you can use ge,le(greaterthan or equalto,lessthan or equalto)

please refer to below link for expression language usage.

https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#lt

[#3]

Hi Shu,

Tanx for replying me, My actual problem is I have files more than 200 and My file size differ from 100 MB to 4 GB. and i need to split those in to 4 Concurrent Processors. could u give some suggestion for this ? the real Problem is i cant route based on file size, i need to send like my first flow file should go to first processor, and second go to second processor, and third go to third processor and finally fourth go to fourth processor, then finally 5'th should go to again first processor and this loop should continue till my last file.

[#4]

@Manikandan Jeyabal

For this case we can use UpdateAttribute processor to add some sequence numbers to the flowfiles then use RouteonAttribute processor to route the flowfiles to one of four processors based on the sequence number.

UpateAttribute Configs:-

seq

${getStateValue('seq'):plus(1)}

so we are adding new property in update attribute processor and get the state value of seq attribute if the value is not present then it gets value as 0 and we are adding 1 to it.

So when first flowfile passes througth the update attribute processor then it get seq attribute with value 1,second flowfile gets 2 value for seq attribute.

Now once the processor reaches to 4 as seq attribute value, then we need to reset that value as 1 again(i.e for 5 flowfile the seq value would be 1 again)

To acheive this we need to use updateattribute processor advance usage and create new rule overthere as

1. Right Click on UpdateAttribute processor
2. Click on Advanced
3. then add new rule in the processor as shown in the below screenshot.

Rules

reset

Conditions

RuleName

reset

Expression

${getStateValue('seq'):equals(4)}

Actions

Attribute

seq
Attribute Value

1

So now the processor will reset the seq attribute value to 1 for every fifth flowfile.

Then Use RouteOnattribute processor:-

So now we are checking the seq attribute value and transfer the first flowfile to first processor and second to second processors..

[#5]

Hi Shu,

that Works Great. Tanx for your sollution.

🙂