Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

In Nifi Tailing multiple directories with the same filename

avatar
Expert Contributor

I am attempting to use TailFile processor to retrieve files from multiple directories with the filename having the same name.

For example:

/var/foo/test1/test.log

/var/foo/test2/test.log

I want to get both of the test.log files.

I have tried using Multiple File mode with /var/foo/*/test.log but that seems to return an error. Is there a way to do this?

Thanks

1 ACCEPTED SOLUTION

avatar
Master Guru

Hi @Eric Lloyd, can you once try to configure processor as

40797-tailtofile.png

what i have tried is:-

i have created 2 directories in my

/tmp

as test1 and test2

and did

echo "test">/tmp/test1/test.log
echo "test">/tmp/test2/test.log

and in the processor i configured Files to tail property as

test[1-2]/test.log

it will look for both test1 and test2 directories

Rolling filename strategy(not required) as test.log as my file names are always test.log

Base directory is /tmp

i mentioned as Fixed name as Rolling strategy and Recursive lookup true

40798-taillogfile.png

As this processor picks up new data and stores the state for each directories.

View solution in original post

3 REPLIES 3

avatar
Master Guru

Hi @Eric Lloyd, can you once try to configure processor as

40797-tailtofile.png

what i have tried is:-

i have created 2 directories in my

/tmp

as test1 and test2

and did

echo "test">/tmp/test1/test.log
echo "test">/tmp/test2/test.log

and in the processor i configured Files to tail property as

test[1-2]/test.log

it will look for both test1 and test2 directories

Rolling filename strategy(not required) as test.log as my file names are always test.log

Base directory is /tmp

i mentioned as Fixed name as Rolling strategy and Recursive lookup true

40798-taillogfile.png

As this processor picks up new data and stores the state for each directories.

avatar
Expert Contributor

Thanks for the answer but maybe I should have clarified. This solution works for my testing environment but my final environment is actually monitoring the versions directory... so it won't always be test1 and test1 but rather

../versions/123.1

.../versions/234.2

and then a new version will be added that is an unknown number. So that is why I was hoping to use a wildcard like: *

Can I use a wildcard? It seems you are using an expression language, maybe with /versions/[*]

avatar
Master Guru

@Eric Lloyd, for this case make we cannot mention wild cards like [*] as this processor wont accepting those regex.

change FilestoTail property to

test[1-2]/[\d|a-z.*]{1,}/test.log

Expression Explanation:-

test[1-2] --look for  test1 or test2 

Then property expression check for

[\d|a-z.*]{1,} --check for directory name as having digits or letters more than one time 

and lists all those directories recursively.

Configs:-

40820-tailtofile.png

For your case:-

Files toTail property should be some thing like below

versions/[\d|a-z.*]{1,}/<your-log-file-name>