I'm working on a data processing flow using Apache NiFi, and I have a scenario that I'd like some guidance on.
I have an Execute SQL script that retrieves data from a table in a SQL Server. Among the data returned, I have source and target folder paths.
My goal is to loop through each of these source folder paths. For each folder, if there are any files present, I need to:
Insert a record into the database to log the file's details.
Move the file from the source folder to the corresponding target folder.
I'm currently using NiFi, and I'm wondering what processors and strategies I should consider to achieve this workflow efficiently. Any insights, recommendations, or examples of similar workflows would be greatly appreciated.