Support Questions
Find answers, ask questions, and share your expertise

NiFi TailFile - Ascedning Rolling Log Files Config Help


Lately I have been working on trying to ingest a set of rolling log files from a Unix machine using NiFi 1.7.1. I've gone through the documentation and what examples I could find but many of it is confusing and oversimplified. My set of logs and how the rotate is a bit awkward to deal with (i do not control it).

Scenario: I have a folder of logs called /opt/logs. In this folder I have several similarly named log files:




Every 15 min the logs are rolled over by renaming them with an ascending numerical value. In this first rotations case it would turn the directory contents into:

/opt/logs/application_george.log (NEW FILE)

/opt/logs/application_george.1.log (OLD FILE - FIRST)

/opt/logs/application_alex.log (NEW FILE)

/opt/logs/application_alex.1.log (OLD FILE - FIRST)

In the second rotations case it would turn the directory contents into:

/opt/logs/application_george.log (NEW FILE)

/opt/logs/application_george.1.log (OLD FILE - SECOND)

/opt/logs/application_george.2.log (OLD FILE - FIRST)

/opt/logs/application_alex.log (NEW FILE)

/opt/logs/application_alex.1.log (OLD FILE - SECOND)

/opt/logs/application_alex.2.log (OLD FILE - FIRST)

This will continue until the maximum number of files is reached (lets say 4), after which the last file is deleted (so when it moves to the 5th rotation).

Attached is my TailFile Processor, from NiFi 1.7.1, configuration. I am not sure if I have done it correctly as after a unknown period of time the state of the processor contains file.x.filenames that reflect the regular expression used in the configuration (ie. "application_[^\.]+\.log") which appear to have instances where one of the files in the directory has been re-read; as such causing duplication of events. Sometimes instead the state will show more than one of the same file listed; example:

file.36.filename /opt/logs/application_alex.log

file.19.filename /opt/logs/application_alex.log

file.27.filename /opt/logs/application_alex.log

I am not sure what I am doing wrong, if anything. Any guidance or suggestions are greatly appreciated.

Thank you