Support Questions
Find answers, ask questions, and share your expertise

NiFi/PutHDFS - How to set number of files to store in destination directory

New Contributor

I'm a new junior member of a team using NiFi and have what is probably a basic question, but I've been unable to find an answer. I need to be able to set the number of files to be stored in a destination directory (100, for example) and have it automatically start a new directory for each 100 files, which will be differentiated by naming directories with a timestamp. Can anyone help? Thanks!

1 REPLY 1

Re: NiFi/PutHDFS - How to set number of files to store in destination directory

@Jan Johnson


You will probably need to use some combination of Wait/Notify Processors to maintain a 0-100 count at which point a new directory is made. Outside of that logic, the putHDFS would need to always be operating with the folder name generated dynamically once 100 counter is satisifed.

These links may be helpful:

https://community.hortonworks.com/questions/186215/simple-nifi-160-waitnotify-example.html

http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/

https://danoyoung.blogspot.com/2017/05/kickin-tires-on-nifis-waitnotify.html