Merge Attribute values from FlowFiles

Question

Hello together,

I splitted a flow file for each x value.

With value x I fetch value y and set it to the Attribute.

Is it possible if I merge the FlowFiles together, that the Attribute values can be merged comma-seperated too?

MattWho · Accepted Answer

@Faerballert

Perhaps you clone your flowfile before the mergeContent processors. So whichever relationship you are connecting to your current mergeContent, you drag a second connection containing that same relationship to a parallel notification flow.

Down this parallel flow path you use a replaceText processor to replace the content with the value from the attribute you want to merge. Then you use a mergeContent processor on this path to merge these files using a "," as your delimiter. Then from this mergeContent you do you notification.

You may also want to open an Apache Jira with your use case and desired improvement for the existing mergeContent. The more details the better.

Hope this helps,

Matt

MattWho · Answer

@Faerballert

The NiFi merge based processors only offer the option to "Keep Common Attributes" (keeps on attributes were every merged file has same attributes with same value) or "Keep all Unique Attributes" (same as above, but will also keep attributes that is unique. This means any attribute that exists in 1 or more of the merged FlowFiles were the value assigned to that attribute is the same in cases where attribute was found on more than one FlowFile being merged).

There is no option to merge all attributes creating a comma separated list of unique values. What is the use case for such a n attribute merge need? There is no way to tell which value goes with which chunk of the merged data. Plus if the merged FlowFile were later split, every produced split FlowFile would have all the same FlowFile attributes.

Hope this helps,
Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.