Created 01-15-2025 01:07 AM
I am using NiFi ListSFTP 2.1.0 to scan files in the root directory of an SFTP server. The Listing Strategy is set to Tracking Timestamps, and the Remote Poll Batch Size is set to 5000. There are over 10,000 files on the server, and each file has a timestamp. The processor only outputs 1000 files during its first run, but subsequent runs do not list the remaining files. However, with the same configuration, ListSFTP 1.9.1 works as expected. Why is this happening?
Created 01-15-2025 05:16 AM
@lexieliu
It may be helpful if you can share the complete configuration of your ListSFTP processor including properties and scheduling tabs.
Thanks,
Matt
Created on 01-15-2025 06:13 PM - edited 01-15-2025 06:20 PM
Thanks for your reply, here is my configuration:
I set this processor to be scheduled every minute. When first scheduled, it output a JSON file containing 5000 data entries. However, subsequent runs did not produce any output. The timestamp recorded in the processor's status is the maximum timestamp from the initial 5000 data entries, which does not exceed the current time.
Created on 01-16-2025 06:26 PM - edited 01-16-2025 06:31 PM
It seems that the next scheduled task has not been triggered