question Re: Send the last CSV file from a NiFi flow to a separate processor in Support Questions

Send the last CSV file from a NiFi flow to a separate processor

Mazilinho — Wed, 10 Apr 2024 11:08:16 GMT

Hello!

I have a flow that extracts 100 CSV files from a SFTP sever.

I need to identify the last CSV file in order to continue the flow after all the files are loaded in GCP bucket.

So, let me explain my scenario: I am extracting all 100 files from SFTP and load them in a GCP bucket. However, after the files are loaded, I want to redirect one file (preferably the last one) using the RouteOnAttribute in order to continue the flow for executing another processors.

Can someone help me please?

Re: Send the last CSV file from a NiFi flow to a separate processor

DianaTorres — Wed, 10 Apr 2024 16:03:19 GMT

@Mazilinho Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @mburgess @joseomjr who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Send the last CSV file from a NiFi flow to a separate processor

Mazilinho — Thu, 11 Apr 2024 07:07:52 GMT

Thank you, Diana! Another way to achieve this is to combine all flow files into one, but using MergeContent processor will not work, because Defragment option will fail, and the Bin-Packing option will merge all the flow-files into multiple files, and not only into one.

Re: Send the last CSV file from a NiFi flow to a separate processor

SAMSAL — Sat, 13 Apr 2024 01:25:25 GMT

So how are you determining the last file? Based on what? is a file created\updated date? or is it based on some sort order?