- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
NIFI getFile processor read file corrupted
- Labels:
-
Apache NiFi
Created 04-15-2020 04:48 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
when I'm uploading a file to a certain folder
Nifi processor "get file" read files from this folder before it's finished uploading the file
so the file became corrupted.
Created 04-15-2020 07:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If your GetFile processor is consuming files before they have finished writing there are a few changes that may help:
1. How are files being written in to the directory? The default "File Filter" will ignore files that start with a ".". If it is possible to change how files are being written to the directory, that will solve your issue through a file filter. For example.... writing new files to directory as ".<filename>" and upon successful write does a rename to remove the dot (this is how ssh works). But you can of course setup any file filter that works for you,
2. Assuming the process that is writing files to the directory is always updating the timestamp on the file, you can use the "Minimum File Age" property to prevent the GetFile from consuming a file until the last modified timestamp in the file has not updated for the configured amount of time. This works in most cases, except when there may be long pauses in the write process that exceeds the configured Min File Age time.
Hope this helps,
Matt
Created 04-15-2020 07:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If your GetFile processor is consuming files before they have finished writing there are a few changes that may help:
1. How are files being written in to the directory? The default "File Filter" will ignore files that start with a ".". If it is possible to change how files are being written to the directory, that will solve your issue through a file filter. For example.... writing new files to directory as ".<filename>" and upon successful write does a rename to remove the dot (this is how ssh works). But you can of course setup any file filter that works for you,
2. Assuming the process that is writing files to the directory is always updating the timestamp on the file, you can use the "Minimum File Age" property to prevent the GetFile from consuming a file until the last modified timestamp in the file has not updated for the configured amount of time. This works in most cases, except when there may be long pauses in the write process that exceeds the configured Min File Age time.
Hope this helps,
Matt
