Have enabled retry option in ListSMB and FetchSMB, since for some reason they are not able to list/fetch files in first attempt.
However, on retry the processor lists and fetches the file successfully.
But, the scheduler is skips fews runs.
Example:
Scheduler is set to run every 30 min and let's assume the job first runs at 00:00 AM.
On first run it fails to either list/fetch the file and the retry kicks in and the files is listed and fetched successfully.
But, the next scheduler run which is suppose to be at 00:30 AM and 01:00 AM are skipped.
Would like to know why retry is impacting scheduler.
Thanks.
@nisaar
Just so I am clear in your dataflow setup...
- ListSMB processor configured to use cron scheduling (get scheduled for ru every ~30 mins)
- ListSMB "success relationship" routed via a connection to FetchSMB processor
- FetchSMB configured to use Timer Driven scheduling with:
- Flow looks like this:
The "retry" when set on a relationship controls whether a FlowFile remains on the inbound connection to the processor or gets routed immediately to the destination relationship. The number of retries is how many attempts will be made to reprocess the source FlowFile before finally routing to the destination relationship. You have 2 so that FlowFile will get terminated if it is not successfully fetched after 2 failed attempts.
I am not clear on this statement you made:
On first run it fails to either list/fetch the file and the retry kicks in and the files is listed and fetched successfully.
There is no "retry" on the listSMB. It simply get scheduled to run at the configured run schedule and listed based upon previous stored state and processor property configuration.
1. How are files being added to the source SMB directory?
2. As files are added how is the the last modified timestamp being updated? (if they are being moved to the SMB as an atomic move, the timestamp on the file may not change which can result in ignored files because another file already listed resulted in sate holding a more recent timestamp).
3. What "Listing strategy" are you using? I recommend using "Tracking Entities" to avoid issue from question 2 above.
Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Thanks for your reply.
Flow setup
Its s 4 node Cluster.
ListSMB details
Listing Strategy : No Tracking
FetchSMB Details
Assume the first run starts at 12:30 AM. It usually fails either during ListSMB or FetchSMB with the error shown below.
The retry then starts at 12:32 AM, and that run completes successfully. It is able to list the files and fetch the file from the SMB location without any issue.
However, the next scheduled run at 1:00 AM gets skipped, and at times even the 1:30 AM run is skipped as well.
After one or two skipped runs, the scheduler starts again, but the same pattern repeats: the first attempt fails, the retry succeeds, and then a few subsequent scheduled runs are skipped.
I would like to understand whether enabling retry has any impact on the scheduler behavior.
Note: We are able to connect to SMB using MacOS finder and also from Java Code where we can list and fetch the file without any issue.
Thanks
