Limit number of files fetched by directory

Question

We have a source directory for Getfile which has one thousand subdirectories (i.e. there are one thousand users who each have a Windows share).

Processing issues arise when a user drops several thousand files into their directory. I presume Getfile scans each directory sequentially and when it finds files, it empties the directory (we delete the source file). So when it comes across a directory that has several thousand files or a directory that is being constantly written to, that user effectively shuts out everyone for several minutes or tens of minutes.

What I would like to do when coming across a directory with many files is to pick up say one hundred and then move on to the next directory. This would allow for a more even distribution among users.

There is <path> attribute that distinguishes between users but I'm not sure how to take advantage of that to solve my problem. Thanks in advance for any tips.

MattWho · Answer

@MikeH

Sounds like you are regularly ingesting a considerable number of files fro your local filesystem. Is this a NiFi multi-node cluster or a single standalone instance of NiFi handling this use case?

Both the GetFile and ListFile processors have a "Path Filter" property that takes a Java Regular expression. You could add multiple processors each with a different regex so they each get from a subset of user sub-directories.

You might consider using the ListFile along with FetchFile processors instead of the GetFile processor. The ListFile processor produces zero byte FlowFiles (1 FlowFile for each file listed), this processor is then connected to a FetchFile processor which use attributes set on that source file to fetch the content and add it to the FlowFile. With a NiFi cluster this design approach allows you to redistributed the 0 byte FlowFiles across all nodes in a NiFi cluster so the heavy work of reading in the content and processing each FlowFile is spread across multiple servers(NiFi cluster nodes). With this approach you can also have many ListFile processor all feeding a single FetchFile.

So perhaps you have a regex for all directories starting with A through C in one processor and another processor for D through F, etc...

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.