Consuming file in NiFi from UNIX by case agnostic filename

[#1]

Hi,

I am trying to consume a file from UNIX folder using NiFi FetchFile processor by providing UNIX folder location and the filename.

The FetchFile processor only fetches the file with the matching filename which is not case agnostic. If I provided FetchFile with filename as abc.txt and I drop ABC.TXT in the UNIX location, the FetchFile processor will not pick it up.

Is there any way I can pickup files based on case agnostic filenames?

Note: I tried using the GetFile processor but it only has success relationship which means that GetFile processor will not handle the negative scenario when the file is not present when expected which is not as per my requirement.

Any help will be appreciated.

Thanks

[#2]

Is this UNIX or Linux? What version? What version of NiFI?

What are the permissions of the directory? file?

[#3]

I am consuming the file from UNIX directory using Apache NiFi 1.5. The directory and file has all the required permissions for NiFi to consume it.

[#4]

@Matt Clarke @Timothy Spann @Felix Albani

[#5]

Expression Language for the file. Try {$filename:toLower()}

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_expression-language/content/tolower.html

https://community.hortonworks.com/questions/71763/how-to-provide-the-path-in-nifi-fetchfile-processo...

[#6]

{$filename:toLower()} will only look for filename with lowercase pattern in the UNIX directory. If I provide filename as ABC.txt to fetch file processor, it will always look for abc.txt but I also need to consume files like aBc.Txt|abC.txt|ABC.TXT in this case.

[#7]

@Harish Vaibhav Kali

Not sure how your existing flow is designed, Try with below approaches.

Approach1:

1.GenerateFlowfile //add new properties absolute.path,filename
2.FetchFile //keep files to fetch property value as ${absolute.path}/${filename:toLower()}
3.UpdateAttribute //connect success connection from FetchFile to this processor

Check are u able to pull all the files now irrespective to case sensitive.

Approach2:

Flow:

1.ListFile //list all the files in the desired directory
2.RouteOnAttribute //add new property and check the filename as ${filename:toLower:equals("abc.txt")}
3.FetchFile //connect the defined property relation from RouteOnAttribute processor
4.UpdateAttribute //dummy processor (connect success connection from FetchFile)

By following this way we are listing out all the files from the desired directory and routing only the desired files to FetchFile processor as this processor does actual fetching of the files.

[#8]

@Harish Vaibhav Kali

-

Your best option may be to go back to using GetFile where you can create a fileFilter regex that is case agnostic for abc.txt.
Then feed success from GetFile to a MonitorActivity processor to handle your "negative scenario".

-

Configure your MonitorActivity to "Continually Send Message" to "true" which will trigger a FlowFile to the inactivity relationship at every configured "threshold duration". Treat this as your missing "abc.txt" processing path.

-

The benefit here is that you can also have a path (activity.restored) that let you know when abc.txt was seen after one or more reported periods where it did not show up.

-

Then route "success" to rest fo your flow. This is the relationship abc.txt will always follow when it actually consumed by GetFile.

-

Thank you,

Matt