How do you identify your files as containing CSV data without looking at each file's content?
Does the filename indicate that it contains CSV data?
Assuming all your csv files have a csv filename extension on them, you could use the routeOnAttribute processor to route on files whose filename ends in .csv to your mergeContent processor. All other FlowFiles with a filename not ending in .csv could then be routed elsewhere in your dataflow.
You would add a new custom property as follows to the routeOnAttribute processor:
Each added dynamic property becomes a new relationship for this processor.
Lets say there is no extension, you may be able to use the RouteOnContent processor to look at the content of each FlowFile for an indicator that it is CSV data and route that way. Of course reading content versus evaluating attributes is more expensive operation in terms of resources.
The MergeContent processor has virtual bins where it groups incoming FlowFiles before merging all the FlowFiles assigned to that bin. The Correlation Attribute property provides a way for you to control what FlowFile are put in which bin. FlowFiles are made up of FlowFile Attributes (key value pairs - basically metadata) and FlowFile content (your actual data). You can use various processors (ie. updateAttribute) to add and manipulate FlowFile attributes on a FlowFile. If you configure your MergeContent processor to use a correlation Attribute, NiFi will look for the attribute key you specify and bin files with the exact same value into the same virtual bin. I do not believe this is what you are looking for here to solve your use case.
While there are scripting processor available in NiFi that can be used to execute your won script against a FlowFile, they are designed to operate against one FlowFile at a time. You could maybe use a putFile processor to write your CSV files to disk and the use one of the scripting processors to merge them.
Another option to is to write your own custom NiFi processor that is specifically designed to merge CSV files.
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
If you feel we have successfully answered your question, please mark an answer as accepted.
Thank you,
Matt
