NiFi ListFile/ListSFTP + FetchFile/FetchSFTP issue

Vikas-Nifi — Thu, 26 Sep 2024 10:22:19 GMT

Hi,

I am using NiFi 1.26.0 and having a flow with combination of ListFile and FetchFile.

Basically i have three files in my local directory (test_1.txt, test_2.txt,test_3.txt). I have to fetch only one file (ex:test_1.txt) for further process.

Based on regex listFile list's all three files.

But facing issue in fetch file. Instead of fetching only one file, it tried to fetch all three files listed by listFile processor and giving output as three files instead of one file, but content of all three files are same as content of file mentioned in fetchFile configuration (test_1.txt)

In data provenance could see it is trying to fetch the files which is not mentioned in fetchFile configuration and after fetching the files it modifies the content.

How can i fix this. Please help.

Re: NiFi ListFile/ListSFTP + FetchFile/FetchSFTP issue

MattWho — Thu, 26 Sep 2024 12:22:07 GMT

@Vikas-Nifi

Your dataflow is working as designed.
You have your listFile producing three FlowFiles (1 for each file listed). Each of those FlowFiles the trigger the execution of your FetchFile which you have configured to fetch the content of only one those files.

If you only want to fetch "test_1.txt", you need to either configure the listFile to only list file "test_1.txt" or you need to add a RouteOnAttribute processor between your listFile and FetchFile so that you are only routing the listed FlowFile with ${filename:equals{'test_1.txt')} to the FetchFile and auto-terminating the other listed files.

The first option of only listing the file you want to fetch the content for is the better option unless there is more to your use case then you have shared.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

question NiFi ListFile/ListSFTP + FetchFile/FetchSFTP issue in Support Questions

NiFi ListFile/ListSFTP + FetchFile/FetchSFTP issue

Re: NiFi ListFile/ListSFTP + FetchFile/FetchSFTP issue