Transfer file and create trigger file in the end using NiFi

[#1]

I have scenario where I need to transfer 10+ files from directory A to another server directory B using FTP at 10pm every day. After transferring all the files to directory B I have to create one trigger file in directory B to trigger the downstream process. How can I do this using NiFi ? Problem is creating trigger file in directory B at the end of all files transfer. please suggest.

Thanks,

Mahendra

[#2 · Accepted solution]

@Mahendra Hegde

One possible solution may be to feed the "Success" relationship from the putFTP processor to a MergeContent processor. The MergeContent processor could be configured to merge once it receives all 10 FlowFiles. The single resulting FlowFile from that merge could be used as your trigger file. UpdateAttribute processor could be used to change it name and ReplaceText could be used to replace its content post merge if your trigger files has a specific requirements. Then use another PutFTP processor to send this trigger file to your FTP server.

Thanks,

Matt

[#3]

Thanks for the quick reply @Matt Clarke 🙂

One more query, I may have N number of files, so how can I find out the number of files in directory and ensure that all files reached Mergecontent processor before I proceed further to create trigger file and FTP ?

[#4]

NiFi does not have a processor that will retrieve a count of the number of files in a source FTP server. You could build in a delay to your MergeContent processor to give sufficient time for all files to be transferred to the NiFi server.

Set the "Minimum Number of Entries" to some value larger then you ever expect N to be. Then set "Max Bin Age" to some value (30 mins for example). The age timer starts when the first FlowFile reaches the MergeContent processor. At the configured Max Bin Age, the bin will be forced to merge even if it has not reached the configured Min Num Entries.

Thanks,

Matt