how to use File filter in GetFile processor at Nifi

[#1]

I have lot of log files I need to get some specific file(based on date) for that I use regular expression in GetFile(file filter option).

I have file with name

nifi-app.log

nifi-app_2017-10-23_10.0.log

nifi-app_2017-10-23_20.0.log

nifi-setup.log

nifi-user_2017-10-06.log

In above files I need only nifi-app_2017* files only for I use regular expression ".*app_2017\.log". After I start flow in result no data is flowed. Can anyone help me whether my regular expression is correct or wrong?

[#2]

@Narasimma varman

If you change your regex to this ".*app_2017.*\.log" it should match only the nifi-app_2017 log files.

[#3]

Thanks Wynner

[#4 · Accepted solution]

@Narasimma varman

if you want only nifi-app_2017 log files then

Input Directory property <directory-path>

change the File Filter property to

nifi-app_2017.*\.log

if you want any logs that having app_2017 in the filename then use

.*app_2017.*\.log 

Keep in mind get file processor is configured

KeepsourceFile property is set to be false by default, once it pulls the files then it deletes them on the directory

If you don't want to delete the files from the directory then change property

KeepsourceFile to true

then the processor wont delete the files once they got pulled.

if you set Recursive SubDirectories property to true then make sure nifi having access to your input Directory.

in addition if you want to do any tailing on the logs then follow the below links to how to configure processors to tail the logs

https://community.hortonworks.com/questions/141403/in-nifi-tailing-multiple-directories-with-the-sam...

https://community.hortonworks.com/questions/141502/how-tailfile-works-with-multiple-files.html?child...

[#5]

Thanks Shu I successfully got correct data

[#6]

Why don`t you use a

	nifi-app_${now():format('yyyy-MM-dd')}_* 

Where the output will be nifi-app_2017-10-25_*.

You can also use attributes for you log prefix and then you route on attribute based on the log type.

[#7]

Thanks Adrian Oprea