Aggregation in nifi

Dmitry — Thu, 06 May 2021 12:57:58 GMT

Hello everebody!

Pls i need to your advise.

I need to allows a number of flow-files together into a single flow-file with use general key.

For example, i have two flow-files with next attributes:

manager_name = John Dow

costumer_info = Matt Damon

costumer_location = LA

and

manager_name = John Dow

costumer_info = Tony Stark

costumer_location = LA

Next, I need to recieve single flow-file with count result of the flow-files with general key = John Dow

Finally single flow-file whith attributes:

manager_name = John Dow

total = 2

How i can do it in nifi?

Re: Aggregation in nifi

MattWho — Thu, 06 May 2021 20:21:03 GMT

@Dmitry

Trying to fully understand your use case here...

You have 1 to many FlowFile where you want to merge the content of each of those FlowFiles in to a single FlowFile if all those FlowFile have an attribute "manager_name" with the same value?

If so, the MergeContent [1] or MergeRecord [2] processors could do that for you. You would set the "Correlation Attribute Name" property in those processors the place incoming FlowFiles in to bins based on unique values set for "manager_name" on each incoming FlowFile. Those bins are then merged based on criteria set in other properties on the processor.

The merged output FlowFile will have several new attribute written to it. One of those new attributes is "merge.count" which will contain the count of the number fo input FlowFiles that were merged.

You could use an UpdateAttribute processor to read the value from "merge.count" and "manager_name", then assign those to other attribute named "total" and "general key" is those attribute names are specifically needed.

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.MergeContent/index.html
[2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.MergeRecord/index.html

If you found this information helped with you question, please take a moment to login and click accept on this solution.
Matt

Re: Aggregation in nifi

Dmitry — Fri, 07 May 2021 08:35:06 GMT

Matt, absolutely right

I tried it and it did work. It was so easy

Thanks a lot

question Re: Aggregation in nifi in Support Questions

Aggregation in nifi

Re: Aggregation in nifi

Re: Aggregation in nifi