Tailfile with rotate log files

[#1]

Hi,

I've got another problem with the logfiles linux weblogic. I've tested several cases (see below) . And I don't know how ingest new data without stop the processor, cancel the state and restart Nifi agent.

First question : Is there another way to do ?

Second question : Is it possible to script the stop processor and the clear state ?

Thanks

The TailFile processor properties :

File to Tail /home/wls.log

Rolling Filename Pattern

State File No value set

Initial Start Position Beginning of File

File Location Local

Test 1 :

cat ‘record01’ >> wls.log

The NiFi Flow Data Provenance is OK. The line ‘record01’ appears.

mv wls.log wls.log.01

cat ‘record02’ >> wls.log

The NiFi Flow Data Provenance is KO. The line ‘record02’ doesn’t appear.

The line ‘record02’ recovers only when I stop the processor TailFile, clear state and restart Nifi.

Test 2 :

cat ‘record01’ >> wls.log

The NiFi Flow Data Provenance is OK. The line ‘record01’ appears.

cp wls.log wls.log.01

rm wls.log

cat ‘record02’ >> wls.log

The NiFi Flow Data Provenance is KO. The line ‘record02’ doesn’t appear.

The line ‘record02’ appears only when I stop the processor TailFile, clear state and restart Nifi.

Test 3 :

cat ‘record01’ >> wls.log

The NiFi Flow Data Provenance is OK. The line ‘record01’ appears.

touch empty.txt

cp empty.txt wls.log (inode ws.log doesn’t change)

cat ‘record02’ >> wls.log

The NiFi Flow Data Provenance is KO. The line ‘record02’ doesn’t appear.

The line ‘record02’ appears only when I stop the processor TailFile, clear state and restart Nifi.

[#2]

Not sure to understand.

If you do the following test:

cat ‘record01’ >> /home/wls.log

The NiFi Flow Data Provenance is OK. The line ‘record01’ appears.

mv /home/wls.log /home/wls.log.01

cat ‘record02’ >> /home/wls.log

I believe the parameters should be the following:

File to Tail /home/wls.log

Rolling Filename Pattern : wls.log.01

Is that what you did?

The observation you made are valid because the processor keeps in its state the position of the last read bytes in the tailed file. This is why if the roll over is not detected by the processor, it won't pick up new data or it will but partially (from the last known position).

[#3]

No it's not what I did. I'm going to test again with Rolling Filename Pattern : wls.log.*.

[#4]

@Pierre Villard

Results are the same even with the new pattern "wls.log.*" and also with all the value of "initial start position".

I think there's a problem.

All I want is to receive (tail) the new records that are written in the wls.log file. And also when the log file rotates and keeps the same name "wls.log".

This case seems very simple yet. But I can't achieve it with Nifi.

So I think there's a little bug. Don't you think the same ?

Please let me know

Thanks