Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How does NiFi handle large files still being written to a directory it is monitoring?

avatar
New Contributor

How does NiFi work for files that take secs/10s of secs/mins to be completely written to a directory (i.e. ops system FTPs file to landing directory NiFi is monitoring)? Does it "stream" the file as its written to? Does it some how know the file is done being written to before it "GetFile"s it?

1 ACCEPTED SOLUTION

avatar
@Brad Surdick

GetFile does not stream the file as it is being written. If you do not configure the GetFile processor correctly, it will pull the incomplete file multiple times. To prevent this from happening, configure the GetFile processor property Minimum File Age to a value, say 30 seconds. The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored.

5870-efbn6.png

View solution in original post

1 REPLY 1

avatar
@Brad Surdick

GetFile does not stream the file as it is being written. If you do not configure the GetFile processor correctly, it will pull the incomplete file multiple times. To prevent this from happening, configure the GetFile processor property Minimum File Age to a value, say 30 seconds. The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored.

5870-efbn6.png