tailfile logfile formant WD-YYYYMMDD.log

[#1]

Hello experts,

I am trying to copy the files being written using tailfile-> putfile. Format of files are below.

WD-YYYYMMDD.log

At any point of time latest file will be get written(ex. WD-20160908.log) and next it will be switched to new date (ex. WD-20160909.log)and that file start getting written.

File to tail : /root/wd/log1/WD-20160831.log

Rolling FileName Pattern: WD-*.log

State of file:

Initial start position: Beginning of File

File location : local

Results are not looks correct. Please suggested.

At source :

[root@server1 log1]# cat > WD-20160830.log
1
2
3
^C
[root@server1 log1]# cat >> WD-20160831.log
4
5
6
^C
[root@server1 log1]# ls -ltr
total 8
-rw-r--r--. 1 root root 6 Sep  9 00:53 WD-20160830.log
-rw-r--r--. 1 root root 6 Sep  9 00:54 WD-20160831.log


At target:


[root@server1 log2]# ls -tlr
total 12
-rw-r--r--. 1 nifi hadoop 2 Sep  9 00:53 WD-20160830.log
-rw-r--r--. 1 nifi hadoop 4 Sep  9 00:54 WD-20160831.0-4.log
-rw-r--r--. 1 nifi hadoop 2 Sep  9 00:54 WD-20160831.4-6.log
[root@*****log2]# cat WD-20160830.log
1
[root@server1 log2]# cat WD-20160831.0-4.log
4
5
[root@server1 log2]# cat WD-20160831.4-6.log
6

[#2 · Accepted solution]

Hi,

At this moment this kind of scenario is not really supported. The processor expects that log messages are appended to a file with a fixed name. Something like:

- /my/var/log/my-app.log

- /my/var/log/my-app.log.1

- /my/var/log/my-app.log.2

Messages being appended to 'my-app.log'.

A workaround would be to update your processor every day with a script/crontab to change the filename and rolling filename properties.

There is a PR to support the scenario you are describing [1] and it is currently in a review process. You can give it a try if you are in position to, otherwise it will probably make it in the next version of NiFi (1.1.0 I think).

[1] https://github.com/apache/nifi/pull/980