Lately I have been working on trying to ingest a set of rolling log files from a Unix machine using NiFi 1.7.1. I've gone through the documentation and what examples I could find but many of it is confusing and oversimplified. My set of logs and how the rotate is a bit awkward to deal with (i do not control it).
Scenario: I have a folder of logs called /opt/logs. In this folder I have several similarly named log files:
Every 15 min the logs are rolled over by renaming them with an ascending numerical value. In this first rotations case it would turn the directory contents into:
This will continue until the maximum number of files is reached (lets say 4), after which the last file is deleted (so when it moves to the 5th rotation).
Attached is my TailFile Processor, from NiFi 1.7.1, configuration. I am not sure if I have done it correctly as after a unknown period of time the state of the processor contains file.x.filenames that reflect the regular expression used in the configuration (ie. "application_[^\.]+\.log") which appear to have instances where one of the files in the directory has been re-read; as such causing duplication of events. Sometimes instead the state will show more than one of the same file listed; example:
I am not sure what I am doing wrong, if anything. Any guidance or suggestions are greatly appreciated.