Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi/PutHDFS - How to set number of files to store in destination directory

NiFi/PutHDFS - How to set number of files to store in destination directory

New Contributor

I'm a new junior member of a team using NiFi and have what is probably a basic question, but I've been unable to find an answer. I need to be able to set the number of files to be stored in a destination directory (100, for example) and have it automatically start a new directory for each 100 files, which will be differentiated by naming directories with a timestamp. Can anyone help? Thanks!

1 REPLY 1
Highlighted

Re: NiFi/PutHDFS - How to set number of files to store in destination directory

Master Collaborator
@Jan Johnson


You will probably need to use some combination of Wait/Notify Processors to maintain a 0-100 count at which point a new directory is made. Outside of that logic, the putHDFS would need to always be operating with the folder name generated dynamically once 100 counter is satisifed.

These links may be helpful:

https://community.hortonworks.com/questions/186215/simple-nifi-160-waitnotify-example.html

http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/

https://danoyoung.blogspot.com/2017/05/kickin-tires-on-nifis-waitnotify.html

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven @ DFHZ

Don't have an account?
Coming from Hortonworks? Activate your account here