Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to use File filter in GetFile processor at Nifi

Solved Go to solution

how to use File filter in GetFile processor at Nifi

New Contributor

I have lot of log files I need to get some specific file(based on date) for that I use regular expression in GetFile(file filter option).

I have file with name

nifi-app.log

nifi-app_2017-10-23_10.0.log

nifi-app_2017-10-23_20.0.log

nifi-setup.log

nifi-user_2017-10-06.log

In above files I need only nifi-app_2017* files only for I use regular expression ".*app_2017\.log". After I start flow in result no data is flowed. Can anyone help me whether my regular expression is correct or wrong?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to use File filter in GetFile processor at Nifi

Super Guru

@Narasimma varman

if you want only nifi-app_2017 log files then

Input Directory property <directory-path>

change the File Filter property to

nifi-app_2017.*\.log

if you want any logs that having app_2017 in the filename then use

.*app_2017.*\.log 

Keep in mind get file processor is configured

KeepsourceFile property is set to be false by default, once it pulls the files then it deletes them on the directory

If you don't want to delete the files from the directory then change property

KeepsourceFile to true

then the processor wont delete the files once they got pulled.

if you set Recursive SubDirectories property to true then make sure nifi having access to your input Directory.

41507-nifilog.png

in addition if you want to do any tailing on the logs then follow the below links to how to configure processors to tail the logs

https://community.hortonworks.com/questions/141403/in-nifi-tailing-multiple-directories-with-the-sam...

https://community.hortonworks.com/questions/141502/how-tailfile-works-with-multiple-files.html?child...

6 REPLIES 6

Re: how to use File filter in GetFile processor at Nifi

@Narasimma varman

If you change your regex to this ".*app_2017.*\.log" it should match only the nifi-app_2017 log files.

Re: how to use File filter in GetFile processor at Nifi

New Contributor

Thanks Wynner

Re: how to use File filter in GetFile processor at Nifi

Super Guru

@Narasimma varman

if you want only nifi-app_2017 log files then

Input Directory property <directory-path>

change the File Filter property to

nifi-app_2017.*\.log

if you want any logs that having app_2017 in the filename then use

.*app_2017.*\.log 

Keep in mind get file processor is configured

KeepsourceFile property is set to be false by default, once it pulls the files then it deletes them on the directory

If you don't want to delete the files from the directory then change property

KeepsourceFile to true

then the processor wont delete the files once they got pulled.

if you set Recursive SubDirectories property to true then make sure nifi having access to your input Directory.

41507-nifilog.png

in addition if you want to do any tailing on the logs then follow the below links to how to configure processors to tail the logs

https://community.hortonworks.com/questions/141403/in-nifi-tailing-multiple-directories-with-the-sam...

https://community.hortonworks.com/questions/141502/how-tailfile-works-with-multiple-files.html?child...

Re: how to use File filter in GetFile processor at Nifi

New Contributor

Thanks Shu I successfully got correct data

Highlighted

Re: how to use File filter in GetFile processor at Nifi

Contributor

Why don`t you use a

	nifi-app_${now():format('yyyy-MM-dd')}_* 

Where the output will be nifi-app_2017-10-25_*.


You can also use attributes for you log prefix and then you route on attribute based on the log type.

Re: how to use File filter in GetFile processor at Nifi

New Contributor

Thanks Adrian Oprea

Don't have an account?
Coming from Hortonworks? Activate your account here