NIFI : tailfile and compressed files

[#1]

We are migrating all our logstash agents to NIFI dataflows.

We encounter a problem with weblogic log (wls.log) : we use ‘tailfile’ processor to ingest; we have a local script that compress once a day this log. At this time, the nifi process uses 200% of CPU, the inode doesn’t change and the file becomes wls.log.gz. Whener, nifi doesn’t bind to the new wls.log. How to avoid it ?

[#2 · Accepted solution]

Hi @David Cuny , I see 2 issues with what you are trying to do:

1. TailFile currently doesn't support logs which got compressed on rollover.
2. TailFile docs explicitly claim to not use 0 seconds as a runtime schedule due to CPU overload, change to a few seconds instead.

Please see https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.TailFile/index...

[#3]

Hi thanks for the answer.

To be more precise, our wls.log is renamed (with timestamp extension) then compressed... I don't know if it changes something ??

For the CPU overload, the runtime schedule is setted to 1. Is it not enough ?

[#4]

Compressed rolled files aren't supported today, this is called out in the docs.

As for runtime schedule - what are the time units? 1 of what? 🙂 Try bumping it more, without having access to a system and flow it's hard to suggest more specific numbers.

[#5]

thx a lot.