I'm a new junior member of a team using NiFi and have what is probably a basic question, but I've been unable to find an answer. I need to be able to set the number of files to be stored in a destination directory (100, for example) and have it automatically start a new directory for each 100 files, which will be differentiated by naming directories with a timestamp. Can anyone help? Thanks!
You will probably need to use some combination of Wait/Notify Processors to maintain a 0-100 count at which point a new directory is made. Outside of that logic, the putHDFS would need to always be operating with the folder name generated dynamically once 100 counter is satisifed.
These links may be helpful:
If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.
Steven @ DFHZ