I'm a new junior member of a team using NiFi and have what is probably a basic question, but I've been unable to find an answer. I need to be able to set the number of files to be stored in a destination directory (100, for example) and have it automatically start a new directory for each 100 files, which will be differentiated by naming directories with a timestamp. Can anyone help? Thanks!
You will probably need to use some combination of Wait/Notify Processors to maintain a 0-100 count at which point a new directory is made. Outside of that logic, the putHDFS would need to always be operating with the folder name generated dynamically once 100 counter is satisifed.
These links may be helpful: