NIFI ListSFTP 2.1.0 only run once

3 replies
1911 views
Translate All

I am using NiFi ListSFTP 2.1.0 to scan files in the root directory of an SFTP server. The Listing Strategy is set to Tracking Timestamps, and the Remote Poll Batch Size is set to 5000. There are over 10,000 files on the server, and each file has a timestamp. The processor only outputs 1000 files during its first run, but subsequent runs do not list the remaining files. However, with the same configuration, ListSFTP 1.9.1 works as expected. Why is this happening?

Apache NiFi

MattWho

Titanium ⭐️⭐️⭐️

@lexieliu
It may be helpful if you can share the complete configuration of your ListSFTP processor including properties and scheduling tabs.

Thanks,
Matt

Translate

lexieliuAuthor

Silver ⭐️⭐️

Thanks for your reply, here is my configuration:

I set this processor to be scheduled every minute. When first scheduled, it output a JSON file containing 5000 data entries. However, subsequent runs did not produce any output. The timestamp recorded in the processor's status is the maximum timestamp from the initial 5000 data entries, which does not exceed the current time.

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.