How does NiFi handle large files still being written to a directory it is monitoring?

surdick311 — Wed, 20 Jul 2016 00:34:18 GMT

How does NiFi work for files that take secs/10s of secs/mins to be completely written to a directory (i.e. ops system FTPs file to landing directory NiFi is monitoring)? Does it "stream" the file as its written to? Does it some how know the file is done being written to before it "GetFile"s it?

Re: How does NiFi handle large files still being written to a directory it is monitoring?

Wynner — Mon, 19 Aug 2019 08:51:29 GMT

@Brad Surdick

GetFile does not stream the file as it is being written. If you do not configure the GetFile processor correctly, it will pull the incomplete file multiple times. To prevent this from happening, configure the GetFile processor property Minimum File Age to a value, say 30 seconds. The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored.

question How does NiFi handle large files still being written to a directory it is monitoring? in Archives of Support Questions (Read Only)

How does NiFi handle large files still being written to a directory it is monitoring?

Re: How does NiFi handle large files still being written to a directory it is monitoring?