Member since
06-18-2024
3
Posts
0
Kudos Received
0
Solutions
06-18-2024
01:47 PM
That's a good idea, however low latency is a user requirement. Currently, processing each file from source to destination takes around one minute. If I add a two minute delay, the users would not be happy.
... View more
06-18-2024
01:45 PM
Thanks Matt, I will look into these ideas. Unfortunately it is all on one server with one NiFi instance. Since these are Windows shares, I am looking at restricting the SMB transfer rate but again I only want to slow down the thousand file guy so we'll see.
... View more
06-18-2024
12:07 PM
We have a source directory for Getfile which has one thousand subdirectories (i.e. there are one thousand users who each have a Windows share). Processing issues arise when a user drops several thousand files into their directory. I presume Getfile scans each directory sequentially and when it finds files, it empties the directory (we delete the source file). So when it comes across a directory that has several thousand files or a directory that is being constantly written to, that user effectively shuts out everyone for several minutes or tens of minutes. What I would like to do when coming across a directory with many files is to pick up say one hundred and then move on to the next directory. This would allow for a more even distribution among users. There is <path> attribute that distinguishes between users but I'm not sure how to take advantage of that to solve my problem. Thanks in advance for any tips.
... View more
Labels:
- Labels:
-
Apache NiFi