question how to use File filter in GetFile processor at Nifi in Archives of Support Questions (Read Only)

how to use File filter in GetFile processor at Nifi

narasimha_varma — Tue, 24 Oct 2017 18:19:26 GMT

I have lot of log files I need to get some specific file(based on date) for that I use regular expression in GetFile(file filter option).

I have file with name

nifi-app.log

nifi-app_2017-10-23_10.0.log

nifi-app_2017-10-23_20.0.log

nifi-setup.log

nifi-user_2017-10-06.log

In above files I need only nifi-app_2017* files only for I use regular expression ".*app_2017\.log". After I start flow in result no data is flowed. Can anyone help me whether my regular expression is correct or wrong?

Re: how to use File filter in GetFile processor at Nifi

Wynner — Tue, 24 Oct 2017 21:51:56 GMT

@Narasimma varman

If you change your regex to this ".*app_2017.*\.log" it should match only the nifi-app_2017 log files.

Re: how to use File filter in GetFile processor at Nifi

Shu_ashu — Sun, 18 Aug 2019 01:25:33 GMT

@Narasimma varman

if you want only nifi-app_2017 log files then

Input Directory property <directory-path>

change the File Filter property to

nifi-app_2017.*\.log

if you want any logs that having app_2017 in the filename then use

.*app_2017.*\.log

Keep in mind get file processor is configured

KeepsourceFile property is set to be false by default, once it pulls the files then it deletes them on the directory

If you don't want to delete the files from the directory then change property

KeepsourceFile to true

then the processor wont delete the files once they got pulled.

if you set Recursive SubDirectories property to true then make sure nifi having access to your input Directory.

in addition if you want to do any tailing on the logs then follow the below links to how to configure processors to tail the logs

https://community.hortonworks.com/questions/141403/in-nifi-tailing-multiple-directories-with-the-same.html?childToView=141479#comment-141479

https://community.hortonworks.com/questions/141502/how-tailfile-works-with-multiple-files.html?childToView=141517#answer-141517

Re: how to use File filter in GetFile processor at Nifi

opreaadrian1 — Wed, 25 Oct 2017 05:23:57 GMT

Why don`t you use a

	nifi-app_${now():format('yyyy-MM-dd')}_*

Where the output will be nifi-app_2017-10-25_*.

You can also use attributes for you log prefix and then you route on attribute based on the log type.

Re: how to use File filter in GetFile processor at Nifi

narasimha_varma — Wed, 25 Oct 2017 14:40:03 GMT

Thanks Shu I successfully got correct data

Re: how to use File filter in GetFile processor at Nifi

narasimha_varma — Wed, 25 Oct 2017 14:51:58 GMT

Thanks Wynner

Re: how to use File filter in GetFile processor at Nifi

narasimha_varma — Wed, 25 Oct 2017 14:52:30 GMT

Thanks Adrian Oprea