Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Aggregation in nifi

avatar
New Contributor

Hello everebody!

 

Pls i need to your advise. 

I need to allows a number of flow-files together into a single flow-file with use general key.

For example, i have two flow-files with next attributes:

manager_name = John Dow

costumer_info = Matt Damon

costumer_location = LA

 

and

 

manager_name = John Dow

costumer_info = Tony Stark

costumer_location = LA

 

Next, I need to recieve single flow-file with count result of the flow-files with general key = John Dow

Finally single flow-file whith attributes:

manager_name = John Dow

total = 2

 

How i can do it in nifi?

 

1 ACCEPTED SOLUTION

avatar
Super Mentor

@Dmitry 

Trying to fully understand your use case here...

You have 1 to many FlowFile where you want to merge the content of each of those FlowFiles in to a single FlowFile if all those FlowFile have an attribute "manager_name" with the same value?

If so, the MergeContent [1] or MergeRecord [2] processors could do that for you.  You would set the "Correlation Attribute Name" property in those processors the place incoming FlowFiles in to bins based on unique values set for "manager_name" on each incoming FlowFile.  Those bins are then merged based on criteria set in other properties on the processor.

The merged output FlowFile will have several new attribute written to it.  One of those new attributes is "merge.count" which will contain the count of the number fo input FlowFiles that were merged.  

You could use an UpdateAttribute processor to read the value from "merge.count" and "manager_name", then assign those to other attribute named "total" and "general key" is those attribute names are specifically needed.

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...
[2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

 

If you found this information helped with you question, please take a moment to login and click accept on this solution.
Matt

View solution in original post

2 REPLIES 2

avatar
Super Mentor

@Dmitry 

Trying to fully understand your use case here...

You have 1 to many FlowFile where you want to merge the content of each of those FlowFiles in to a single FlowFile if all those FlowFile have an attribute "manager_name" with the same value?

If so, the MergeContent [1] or MergeRecord [2] processors could do that for you.  You would set the "Correlation Attribute Name" property in those processors the place incoming FlowFiles in to bins based on unique values set for "manager_name" on each incoming FlowFile.  Those bins are then merged based on criteria set in other properties on the processor.

The merged output FlowFile will have several new attribute written to it.  One of those new attributes is "merge.count" which will contain the count of the number fo input FlowFiles that were merged.  

You could use an UpdateAttribute processor to read the value from "merge.count" and "manager_name", then assign those to other attribute named "total" and "general key" is those attribute names are specifically needed.

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...
[2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

 

If you found this information helped with you question, please take a moment to login and click accept on this solution.
Matt

avatar
New Contributor

Matt, absolutely right

I tried it and it did work. It was so easy

 

Thanks a lot