question Re: Need info on Regex to be used in NiFi ListFile processor in Support Questions

Need info on Regex to be used in NiFi ListFile processor

Rohitravi — Mon, 03 Feb 2020 13:45:00 GMT

Hi All,

I have directory structure like below on Unix file system,
/DIR1/DIR2/DIR3/DIR4/DIR5

I have files under the directories DIR3 & DIR5. However I want to read files only from DIR5.
I tried with various regex but could not get the right one

Could you please help me with the Regex to be used in ListFile processor to read files from DIR5 only.

Thanks in Advance!

Re: Need info on Regex to be used in NiFi ListFile processor

MattWho — Mon, 03 Feb 2020 22:17:52 GMT

@Rohitravi

If you only want to list files from directory DIR5, simply provide the complete path to DIR5 in the "Input Directory" ListFile processor configuration property and set Recurse Subdirectories as false.

If above is not an option, you may want to try using the following "Path Filter":

.*?/DIR5

Hope this helps,

Matt

Re: Need info on Regex to be used in NiFi ListFile processor

Rohitravi — Tue, 04 Feb 2020 05:14:50 GMT

@MattWho

Hi Matt,

Thanks for your response.

I tried the regex, but it picks the file which is present under DIR3 also. Below is my configuration,

Re: Need info on Regex to be used in NiFi ListFile processor

MattWho — Tue, 04 Feb 2020 20:43:48 GMT

@Rohitravi

The Path Filter is applied against all subdirectories of the configured "Input Directory". Any files found in the base "Input Directory" are still going to be listed. If you had files in "dir4", they should not have been listed.

Is dir2 empty? If so can you change your "Input Directory" to /dir1/dir2 instead of /dir1/dir2/dir3.

I cannot think of a reason why when filtering based on subdir path that you would still expect to returns from the base directory, so I filed an Apache jira ( https://issues.apache.org/jira/browse/NIFI-7104 ).

Another option is to add a RouteOnAttribute processor after your listFile processor to route on only FlowFile where the absolute.path FlowFile attribute included "dir5".

Then auto-terminate the unmatched relationship and route the "dir5" relationship on to the next component in your dataflow.

Hope this helps,

Matt

If you found this solution resolves your query, please take a moment to click accept.

Re: Need info on Regex to be used in NiFi ListFile processor

Rohitravi — Wed, 05 Feb 2020 04:34:49 GMT

Thanks a lot Matt!!