question Re: NIFI ListSFTP 2.1.0 only run once in Support Questions

NIFI ListSFTP 2.1.0 only run once

lexieliu — Wed, 15 Jan 2025 09:07:25 GMT

I am using NiFi ListSFTP 2.1.0 to scan files in the root directory of an SFTP server. The Listing Strategy is set to Tracking Timestamps, and the Remote Poll Batch Size is set to 5000. There are over 10,000 files on the server, and each file has a timestamp. The processor only outputs 1000 files during its first run, but subsequent runs do not list the remaining files. However, with the same configuration, ListSFTP 1.9.1 works as expected. Why is this happening?

Re: NIFI ListSFTP 2.1.0 only run once

MattWho — Wed, 15 Jan 2025 13:16:59 GMT

@lexieliu
It may be helpful if you can share the complete configuration of your ListSFTP processor including properties and scheduling tabs.

Thanks,
Matt

Re: NIFI ListSFTP 2.1.0 only run once

lexieliu — Thu, 16 Jan 2025 02:20:03 GMT

Thanks for your reply, here is my configuration:

I set this processor to be scheduled every minute. When first scheduled, it output a JSON file containing 5000 data entries. However, subsequent runs did not produce any output. The timestamp recorded in the processor's status is the maximum timestamp from the initial 5000 data entries, which does not exceed the current time.

Re: NIFI ListSFTP 2.1.0 only run once

lexieliu — Fri, 17 Jan 2025 02:31:31 GMT

It seems that the next scheduled task has not been triggered