Have enabled retry option in ListSMB and FetchSMB, since for some reason they are not able to list/fetch files in first attempt.
However, on retry the processor lists and fetches the file successfully.
But, the scheduler is skips fews runs.
Example:
Scheduler is set to run every 30 min and let's assume the job first runs at 00:00 AM.
On first run it fails to either list/fetch the file and the retry kicks in and the files is listed and fetched successfully.
But, the next scheduler run which is suppose to be at 00:30 AM and 01:00 AM are skipped.
Would like to know why retry is impacting scheduler.
Thanks.
@nisaar
I expect that retry set on the Success relationship out of ListSMB is impacting your scheduling. I suspect that retry is blocking "successful" attempts by until both retry have been made which aligns with the skip twice you are seeing in scheduling. As I mentioned before, you should not be setting "retry" on any success relationships. This is an anti-pattern that will delay processing of any successful execution for the duration of the retry (Each retry would occur at the processors scheduled execution times).
Note: Scheduling of a NiFi component does not mean immediate execution. That execution depends on availability of threads to service the execution. NiFi has a "max timer driven thread count" configuration that establishes the thread pool from which all scheduled component threads come from.
So things like number of running components, number of concurrent tasks set on a given component, CPU insensitive components, etc can impact when a "scheduled" component is given a thread from the thread pool to execute.
I Thread pool does not impact scheduler unless the processor has been scheduled and never executes until within the 1.5 hours since it was initially scheduled (which I doubt in this case). The more logical cause because of the consistent behavior is the "retry" you have set on Success from ListSMB.
Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
