question Re: NIFI : tailfile and compressed files in Archives of Support Questions (Read Only)

NIFI : tailfile and compressed files

david_cuny — Fri, 20 May 2016 15:48:16 GMT

We are migrating all our logstash agents to NIFI dataflows.

We encounter a problem with weblogic log (wls.log) : we use ‘tailfile’ processor to ingest; we have a local script that compress once a day this log. At this time, the nifi process uses 200% of CPU, the inode doesn’t change and the file becomes wls.log.gz. Whener, nifi doesn’t bind to the new wls.log. How to avoid it ?

Re: NIFI : tailfile and compressed files

andrewg — Sat, 21 May 2016 01:52:52 GMT

Hi @David Cuny , I see 2 issues with what you are trying to do:

TailFile currently doesn't support logs which got compressed on rollover.
TailFile docs explicitly claim to not use 0 seconds as a runtime schedule due to CPU overload, change to a few seconds instead.

Please see https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.TailFile/index.html

Re: NIFI : tailfile and compressed files

david_cuny — Tue, 31 May 2016 21:13:38 GMT

Hi thanks for the answer.

To be more precise, our wls.log is renamed (with timestamp extension) then compressed... I don't know if it changes something ??

For the CPU overload, the runtime schedule is setted to 1. Is it not enough ?

Re: NIFI : tailfile and compressed files

andrewg — Tue, 31 May 2016 22:44:52 GMT

Compressed rolled files aren't supported today, this is called out in the docs.

As for runtime schedule - what are the time units? 1 of what? 🙂 Try bumping it more, without having access to a system and flow it's hard to suggest more specific numbers.

Re: NIFI : tailfile and compressed files

david_cuny — Wed, 01 Jun 2016 21:00:24 GMT

thx a lot.