In Nifi Tailing multiple directories with the same filename

[#1]

I am attempting to use TailFile processor to retrieve files from multiple directories with the filename having the same name.

For example:

/var/foo/test1/test.log

/var/foo/test2/test.log

I want to get both of the test.log files.

I have tried using Multiple File mode with /var/foo/*/test.log but that seems to return an error. Is there a way to do this?

Thanks

[#2 · Accepted solution]

Hi @Eric Lloyd, can you once try to configure processor as

what i have tried is:-

i have created 2 directories in my

/tmp

as test1 and test2

and did

echo "test">/tmp/test1/test.log
echo "test">/tmp/test2/test.log

and in the processor i configured Files to tail property as

test[1-2]/test.log

it will look for both test1 and test2 directories

Rolling filename strategy(not required) as test.log as my file names are always test.log

Base directory is /tmp

i mentioned as Fixed name as Rolling strategy and Recursive lookup true

As this processor picks up new data and stores the state for each directories.

[#3]

Thanks for the answer but maybe I should have clarified. This solution works for my testing environment but my final environment is actually monitoring the versions directory... so it won't always be test1 and test1 but rather

../versions/123.1

.../versions/234.2

and then a new version will be added that is an unknown number. So that is why I was hoping to use a wildcard like: *

Can I use a wildcard? It seems you are using an expression language, maybe with /versions/[*]

[#4]

@Eric Lloyd, for this case make we cannot mention wild cards like [*] as this processor wont accepting those regex.

change FilestoTail property to

test[1-2]/[\d|a-z.*]{1,}/test.log

Expression Explanation:-

test[1-2] --look for  test1 or test2 

Then property expression check for

[\d|a-z.*]{1,} --check for directory name as having digits or letters more than one time 

and lists all those directories recursively.

Configs:-

For your case:-

Files toTail property should be some thing like below

versions/[\d|a-z.*]{1,}/<your-log-file-name>