Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to re route flow files to 4 Different processors ?

Solved Go to solution
Highlighted

How to re route flow files to 4 Different processors ?

Contributor

Hi All,

please help me out to solve this issue.

Im having 10 Different flow files in the following size.

3 files size (100-500MB)

2 Files (1GB)

2 Files(1.8GB)

3 files (50MB)

I need to write some logic that should reroute these 10 files to 4 Convert Record processors running parallely ? is there any way that we could do this ?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to re route flow files to 4 Different processors ?

Super Guru

@Manikandan Jeyabal
You can use RouteonAttribute with fileSize attribute and check if the size of flowfile is greater than 1GB.

Example:-

64498-filesize.png

1.filesize > 1gig

 ${fileSize:gt(1048576000)}

in the above routeonattribute processor i have added new property which can detect and transfer flowfiles size if the size is greater than 1GB. we need to keep the size of file in bytes.

2.filesize >100MB and <500MB

${fileSize:gt(1048576):and(${fileSize:lt(524288000) 

second property will route the flowfiles which are having sizes greater than 100MB and less than 500 MB.

Like this way you can add new properties in Routeonattribute and transfer the flowfiles to corresponding convert record processors based on Filesizes.

**1 megabyte (1048576 bytes)**

**1 Gigabyte = 1000 Megabytes**

As i'm using gt,lt(greaterthan,lessthan) based on your requirements you can use ge,le(greaterthan or equalto,lessthan or equalto)

please refer to below link for expression language usage.

https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#lt

4 REPLIES 4

Re: How to re route flow files to 4 Different processors ?

Super Guru

@Manikandan Jeyabal
You can use RouteonAttribute with fileSize attribute and check if the size of flowfile is greater than 1GB.

Example:-

64498-filesize.png

1.filesize > 1gig

 ${fileSize:gt(1048576000)}

in the above routeonattribute processor i have added new property which can detect and transfer flowfiles size if the size is greater than 1GB. we need to keep the size of file in bytes.

2.filesize >100MB and <500MB

${fileSize:gt(1048576):and(${fileSize:lt(524288000) 

second property will route the flowfiles which are having sizes greater than 100MB and less than 500 MB.

Like this way you can add new properties in Routeonattribute and transfer the flowfiles to corresponding convert record processors based on Filesizes.

**1 megabyte (1048576 bytes)**

**1 Gigabyte = 1000 Megabytes**

As i'm using gt,lt(greaterthan,lessthan) based on your requirements you can use ge,le(greaterthan or equalto,lessthan or equalto)

please refer to below link for expression language usage.

https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#lt

Re: How to re route flow files to 4 Different processors ?

Contributor

Hi Shu,

Tanx for replying me, My actual problem is I have files more than 200 and My file size differ from 100 MB to 4 GB. and i need to split those in to 4 Concurrent Processors. could u give some suggestion for this ? the real Problem is i cant route based on file size, i need to send like my first flow file should go to first processor, and second go to second processor, and third go to third processor and finally fourth go to fourth processor, then finally 5'th should go to again first processor and this loop should continue till my last file.

Re: How to re route flow files to 4 Different processors ?

Super Guru
@Manikandan Jeyabal

For this case we can use UpdateAttribute processor to add some sequence numbers to the flowfiles then use RouteonAttribute processor to route the flowfiles to one of four processors based on the sequence number.

UpateAttribute Configs:-

62677-updateattribute-storestate.png

seq

${getStateValue('seq'):plus(1)}

so we are adding new property in update attribute processor and get the state value of seq attribute if the value is not present then it gets value as 0 and we are adding 1 to it.

So when first flowfile passes througth the update attribute processor then it get seq attribute with value 1,second flowfile gets 2 value for seq attribute.

Now once the processor reaches to 4 as seq attribute value, then we need to reset that value as 1 again(i.e for 5 flowfile the seq value would be 1 again)

To acheive this we need to use updateattribute processor advance usage and create new rule overthere as

  1. Right Click on UpdateAttribute processor
  2. Click on Advanced
  3. then add new rule in the processor as shown in the below screenshot.

62678-updateattribute-advance-usage.png

Rules

reset

Conditions
RuleName
reset
Expression
${getStateValue('seq'):equals(4)}
Actions
Attribute
seq
Attribute Value
1
So now the processor will reset the seq attribute value to 1 for every fifth flowfile.
Then Use RouteOnattribute processor:-

62679-routeonattribute.png

So now we are checking the seq attribute value and transfer the first flowfile to first processor and second to second processors..

Re: How to re route flow files to 4 Different processors ?

Contributor

Hi Shu,

that Works Great. Tanx for your sollution.

:)