Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to use File filter in GetFile processor at Nifi

avatar
Rising Star

I have lot of log files I need to get some specific file(based on date) for that I use regular expression in GetFile(file filter option).

I have file with name

nifi-app.log

nifi-app_2017-10-23_10.0.log

nifi-app_2017-10-23_20.0.log

nifi-setup.log

nifi-user_2017-10-06.log

In above files I need only nifi-app_2017* files only for I use regular expression ".*app_2017\.log". After I start flow in result no data is flowed. Can anyone help me whether my regular expression is correct or wrong?

1 ACCEPTED SOLUTION

avatar
Master Guru

@Narasimma varman

if you want only nifi-app_2017 log files then

Input Directory property <directory-path>

change the File Filter property to

nifi-app_2017.*\.log

if you want any logs that having app_2017 in the filename then use

.*app_2017.*\.log 

Keep in mind get file processor is configured

KeepsourceFile property is set to be false by default, once it pulls the files then it deletes them on the directory

If you don't want to delete the files from the directory then change property

KeepsourceFile to true

then the processor wont delete the files once they got pulled.

if you set Recursive SubDirectories property to true then make sure nifi having access to your input Directory.

41507-nifilog.png

in addition if you want to do any tailing on the logs then follow the below links to how to configure processors to tail the logs

https://community.hortonworks.com/questions/141403/in-nifi-tailing-multiple-directories-with-the-sam...

https://community.hortonworks.com/questions/141502/how-tailfile-works-with-multiple-files.html?child...

View solution in original post

6 REPLIES 6

avatar
@Narasimma varman

If you change your regex to this ".*app_2017.*\.log" it should match only the nifi-app_2017 log files.

avatar
Rising Star

Thanks Wynner

avatar
Master Guru

@Narasimma varman

if you want only nifi-app_2017 log files then

Input Directory property <directory-path>

change the File Filter property to

nifi-app_2017.*\.log

if you want any logs that having app_2017 in the filename then use

.*app_2017.*\.log 

Keep in mind get file processor is configured

KeepsourceFile property is set to be false by default, once it pulls the files then it deletes them on the directory

If you don't want to delete the files from the directory then change property

KeepsourceFile to true

then the processor wont delete the files once they got pulled.

if you set Recursive SubDirectories property to true then make sure nifi having access to your input Directory.

41507-nifilog.png

in addition if you want to do any tailing on the logs then follow the below links to how to configure processors to tail the logs

https://community.hortonworks.com/questions/141403/in-nifi-tailing-multiple-directories-with-the-sam...

https://community.hortonworks.com/questions/141502/how-tailfile-works-with-multiple-files.html?child...

avatar
Rising Star

Thanks Shu I successfully got correct data

avatar
Rising Star

Why don`t you use a

	nifi-app_${now():format('yyyy-MM-dd')}_* 

Where the output will be nifi-app_2017-10-25_*.


You can also use attributes for you log prefix and then you route on attribute based on the log type.

avatar
Rising Star

Thanks Adrian Oprea