Aggregation in nifi

Question

Hello everebody!

Pls i need to your advise.

I need to allows a number of flow-files together into a single flow-file with use general key.

For example, i have two flow-files with next attributes:

manager_name = John Dow

costumer_info = Matt Damon

costumer_location = LA

and

manager_name = John Dow

costumer_info = Tony Stark

costumer_location = LA

Next, I need to recieve single flow-file with count result of the flow-files with general key = John Dow

Finally single flow-file whith attributes:

manager_name = John Dow

total = 2

How i can do it in nifi?

MattWho · Accepted Answer

@Dmitry

Trying to fully understand your use case here...

You have 1 to many FlowFile where you want to merge the content of each of those FlowFiles in to a single FlowFile if all those FlowFile have an attribute "manager_name" with the same value?

If so, the MergeContent [1] or MergeRecord [2] processors could do that for you. You would set the "Correlation Attribute Name" property in those processors the place incoming FlowFiles in to bins based on unique values set for "manager_name" on each incoming FlowFile. Those bins are then merged based on criteria set in other properties on the processor.

The merged output FlowFile will have several new attribute written to it. One of those new attributes is "merge.count" which will contain the count of the number fo input FlowFiles that were merged.

You could use an UpdateAttribute processor to read the value from "merge.count" and "manager_name", then assign those to other attribute named "total" and "general key" is those attribute names are specifically needed.

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.MergeContent/index.html
[2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.MergeRecord/index.html

If you found this information helped with you question, please take a moment to login and click accept on this solution.
Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.