question Re: In Nifi Tailing multiple directories with the same filename in Archives of Support Questions (Read Only)

In Nifi Tailing multiple directories with the same filename

elloyd — Thu, 12 Oct 2017 06:09:21 GMT

I am attempting to use TailFile processor to retrieve files from multiple directories with the filename having the same name.

For example:

/var/foo/test1/test.log

/var/foo/test2/test.log

I want to get both of the test.log files.

I have tried using Multiple File mode with /var/foo/*/test.log but that seems to return an error. Is there a way to do this?

Thanks

Re: In Nifi Tailing multiple directories with the same filename

Shu_ashu — Sun, 18 Aug 2019 02:47:58 GMT

Hi @Eric Lloyd, can you once try to configure processor as

what i have tried is:-

i have created 2 directories in my

/tmp

as test1 and test2

and did

echo "test">/tmp/test1/test.log
echo "test">/tmp/test2/test.log

and in the processor i configured Files to tail property as

test[1-2]/test.log

it will look for both test1 and test2 directories

Rolling filename strategy(not required) as test.log as my file names are always test.log

Base directory is /tmp

i mentioned as Fixed name as Rolling strategy and Recursive lookup true

As this processor picks up new data and stores the state for each directories.

Re: In Nifi Tailing multiple directories with the same filename

elloyd — Thu, 12 Oct 2017 23:50:22 GMT

Thanks for the answer but maybe I should have clarified. This solution works for my testing environment but my final environment is actually monitoring the versions directory... so it won't always be test1 and test1 but rather

../versions/123.1

.../versions/234.2

and then a new version will be added that is an unknown number. So that is why I was hoping to use a wildcard like: *

Can I use a wildcard? It seems you are using an expression language, maybe with /versions/[*]

Re: In Nifi Tailing multiple directories with the same filename

Shu_ashu — Sun, 18 Aug 2019 02:47:44 GMT

@Eric Lloyd, for this case make we cannot mention wild cards like [*] as this processor wont accepting those regex.

change FilestoTail property to

test[1-2]/[\d|a-z.*]{1,}/test.log

Expression Explanation:-

test[1-2] --look for  test1 or test2

Then property expression check for

[\d|a-z.*]{1,} --check for directory name as having digits or letters more than one time

and lists all those directories recursively.

Configs:-

For your case:-

Files toTail property should be some thing like below

versions/[\d|a-z.*]{1,}/<your-log-file-name>