Gold ⭐️

Solved

Retry impacts scheduler

17 replies
2880 views
Translate All

Have enabled retry option in ListSMB and FetchSMB, since for some reason they are not able to list/fetch files in first attempt.

However, on retry the processor lists and fetches the file successfully.

But, the scheduler is skips fews runs.

Example:

Scheduler is set to run every 30 min and let's assume the job first runs at 00:00 AM.

On first run it fails to either list/fetch the file and the retry kicks in and the files is listed and fetched successfully.

But, the next scheduler run which is suppose to be at 00:30 AM and 01:00 AM are skipped.

Would like to know why retry is impacting scheduler.

Thanks.

Apache NiFi

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@nisaar

I expect that retry set on the Success relationship out of ListSMB is impacting your scheduling. I suspect that retry is blocking "successful" attempts by until both retry have been made which aligns with the skip twice you are seeing in scheduling. As I mentioned before, you should not be setting "retry" on any success relationships. This is an anti-pattern that will delay processing of any successful execution for the duration of the retry (Each retry would occur at the processors scheduled execution times).

Note: Scheduling of a NiFi component does not mean immediate execution. That execution depends on availability of threads to service the execution. NiFi has a "max timer driven thread count" configuration that establishes the thread pool from which all scheduled component threads come from.

So things like number of running components, number of concurrent tasks set on a given component, CPU insensitive components, etc can impact when a "scheduled" component is given a thread from the thread pool to execute.

I Thread pool does not impact scheduler unless the processor has been scheduled and never executes until within the 1.5 hours since it was initially scheduled (which I doubt in this case). The more logical cause because of the consistent behavior is the "retry" you have set on Success from ListSMB.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

MattWho

Titanium ⭐️⭐️⭐️

@nisaar

Just so I am clear in your dataflow setup...

- ListSMB processor configured to use cron scheduling (get scheduled for ru every ~30 mins)
- ListSMB "success relationship" routed via a connection to FetchSMB processor
- FetchSMB configured to use Timer Driven scheduling with:

- Flow looks like this:

The "retry" when set on a relationship controls whether a FlowFile remains on the inbound connection to the processor or gets routed immediately to the destination relationship. The number of retries is how many attempts will be made to reprocess the source FlowFile before finally routing to the destination relationship. You have 2 so that FlowFile will get terminated if it is not successfully fetched after 2 failed attempts.

I am not clear on this statement you made:

On first run it fails to either list/fetch the file and the retry kicks in and the files is listed and fetched successfully.

There is no "retry" on the listSMB. It simply get scheduled to run at the configured run schedule and listed based upon previous stored state and processor property configuration.

1. How are files being added to the source SMB directory?
2. As files are added how is the the last modified timestamp being updated? (if they are being moved to the SMB as an atomic move, the timestamp on the file may not change which can result in ignored files because another file already listed resulted in sate holding a more recent timestamp).
3. What "Listing strategy" are you using? I recommend using "Tracking Entities" to avoid issue from question 2 above.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

nisaarAuthor

Gold ⭐️

@MattWho

Thanks for your reply.

Flow setup

Its s 4 node Cluster.

ListSMB details

Listing Strategy : No Tracking

FetchSMB Details

Assume the first run starts at 12:30 AM. It usually fails either during ListSMB or FetchSMB with the error shown below.

The retry then starts at 12:32 AM, and that run completes successfully. It is able to list the files and fetch the file from the SMB location without any issue.

However, the next scheduled run at 1:00 AM gets skipped, and at times even the 1:30 AM run is skipped as well.

After one or two skipped runs, the scheduler starts again, but the same pattern repeats: the first attempt fails, the retry succeeds, and then a few subsequent scheduled runs are skipped.

I would like to understand whether enabling retry has any impact on the scheduler behavior.

Note: We are able to connect to SMB using MacOS finder and also from Java Code where we can list and fetch the file without any issue.

Thanks

Translate

MattWho

Titanium ⭐️⭐️⭐️

@nisaar

I see you set retry on the ListSMB processor "success" relationship. You should never set retry on any "Success" relationship. The connection retry logic is triggered when a FlowFile would be routed to the relationship. Instead of routing the FlowFile to the relationship it remains on the upstream connection to be "retried". That means that only when FetchSMB successfully lists a file will it trigger retry. Plus retry implies a there is some upstream location to persist a source FlowFile. Only once all retry attempts are exhausted will the FlowFile actually be placed on the connection containing the "success" relationship. I would expect unexpected behavior with such a configuration.

Using "retry" on a failure or other error type relationship makes sense since these are the relationships you don't expect FlowFiles to be routed to. so makes sense on your FetchSMB processor so that source FlowFile will remain on the inbound connection to FetchSMB until both the configured retry attempts are made. I both retries fail I see you terminate the "Failure" relationship which effectively throws away that file. I don't know that terminate is what i would do here either since ListSMB was successful in order to produce FlowFile waiting at FetchSMB. Plus state is recorded by ListSMB and the file that failed ot be fetched would never get listed again unless you touched file to update last modified time or cleared state on listSMB processor.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

nisaarAuthor

Gold ⭐️

Have removed the Retry on ListSMB

On running ListSMB for first time, I get below error,

ListSmb[id=678b3091-1e8c-38c6-a274-0d2eb00c91cb] Failed to perform listing on remote host due to Could not perform listing: java.io.IOException: Could not perform listing
- Caused by: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.TimeoutException: Timeout expired

On running ListSMB second time, its able to list the files, but FetchSMB fails with below 2 type of errors, not all the files fails though.

FetchSmb[id=6b1f9f93-4d76-38fc-94ed-08918722a36b] Could not fetch file OTL-Prod-Reports/OTL_Manual_Override//Errors_and_Warnings/Events/Events_OTL_Override_Error_Warnings_2026_04_24.csv.: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.TimeoutException: Timeout expired

FetchSmb[id=6b1f9f93-4d76-38fc-94ed-08918722a36b] Could not fetch file OTL-Prod-Reports/OTL_Manual_Override//Errors_and_Warnings/New_Stores/NewStores_OTL_Override_Error_Warnings_2026_04_25.csv.: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: Cannot write SMB2_SESSION_SETUP with message id << 169 >> as transport is disconnected
- Caused by: com.hierynomus.protocol.transport.TransportException: Cannot write SMB2_SESSION_SETUP with message id << 169 >> as transport is disconnected

Translate

nisaarAuthor

Gold ⭐️

Have removed the Retry on ListSMB

On running ListSMB for first time, I get below error,

ListSmb[id=678b3091-1e8c-38c6-a274-0d2eb00c91cb] Failed to perform listing on remote host due to Could not perform listing: java.io.IOException: Could not perform listing
- Caused by: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.TimeoutException: Timeout expired

On running ListSMB second time, its able to list the files, but FetchSMB fails with below 2 type of errors, not all the files fails though.

FetchSmb[id=6b1f9f93-4d76-38fc-94ed-08918722a36b] Could not fetch file OTL-Prod-Reports/OTL_Manual_Override//Errors_and_Warnings/Events/Events_OTL_Override_Error_Warnings_2026_04_24.csv.: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.TimeoutException: Timeout expired

FetchSmb[id=6b1f9f93-4d76-38fc-94ed-08918722a36b] Could not fetch file OTL-Prod-Reports/OTL_Manual_Override//Errors_and_Warnings/New_Stores/NewStores_OTL_Override_Error_Warnings_2026_04_25.csv.: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: Cannot write SMB2_SESSION_SETUP with message id << 169 >> as transport is disconnected
- Caused by: com.hierynomus.protocol.transport.TransportException: Cannot write SMB2_SESSION_SETUP with message id << 169 >> as transport is disconnected

Thanks

Translate

nisaarAuthor

Gold ⭐️

Have removed the Retry on ListSMB

On running ListSMB for first time, I get below error,

ListSmb[id=678b3091-1e8c-38c6-a274-0d2eb00c91cb] Failed to perform listing on remote host due to Could not perform listing: java.io.IOException: Could not perform listing
- Caused by: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.TimeoutException: Timeout expired

On running ListSMB second time, its able to list the files, but FetchSMB fails with below 2 type of errors, not all the files fails though.

FetchSmb[id=6b1f9f93-4d76-38fc-94ed-08918722a36b] Could not fetch file OTL-Prod-Reports/OTL_Manual_Override//Errors_and_Warnings/Events/Events_OTL_Override_Error_Warnings_2026_04_24.csv.: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired
- Caused by: java.util.concurrent.TimeoutException: Timeout expired

FetchSmb[id=6b1f9f93-4d76-38fc-94ed-08918722a36b] Could not fetch file OTL-Prod-Reports/OTL_Manual_Override//Errors_and_Warnings/New_Stores/NewStores_OTL_Override_Error_Warnings_2026_04_25.csv.: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP
- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: Cannot write SMB2_SESSION_SETUP with message id << 169 >> as transport is disconnected
- Caused by: com.hierynomus.protocol.transport.TransportException: Cannot write SMB2_SESSION_SETUP with message id << 169 >> as transport is disconnected

Translate

MattWho

Titanium ⭐️⭐️⭐️

@nisaar

Why are you posting the same response over and over again every few hours?
It makes the thread unnecessarily noisy.

Your issue is a timeout issue within the SMBJClientProviderService that is attempting to establish a connection with your target SMB share. This is taking longer then the configured timeout. The subsequent request is probably working because the connection is still established at time of second run. Same issue with the FetchSMB it appears as the connection is closed at time of attempting to fetch the File content. Have you tried increasing the Timeout setting in that controllers service from the default 5 secs to 20 seconds?

Have you inspected your network latency between NiFi host and SMB server?
Are you seeing any packet loss?

Have you inspected the SMB server logs in Windows?

Does the behavior change if you stop using the "run once" option and just start the processor? Your run schedule of every 30 seconds is allowing the connection to timeout. If you also adjust that run schedule to 10 seconds on ListSMB, do you see a change in behavior?

Unfortunately I do not have a SMB share available to test this setup myself and am providing what guidance and suggestions I can to help with your community query.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

nisaarAuthor

Gold ⭐️

Sorry for reposting again and again, whatever I post was not visible to me on page refresh.

So was confused if its getting posted or not.

- Yes, have tried increasing the timeout upto 120 Secs, still same issue.

- Out storage team don't save the SMB server logs.

- I'm not sure how to check packet loss.

- Yes, have already tried running it based on scheduler instead of "Run Once" and still same error.

- Its scheduled to run every 30 min not 30 seconds.

Translate

MattWho

Titanium ⭐️⭐️⭐️

@Former Member

Do you observe same issue if you run it more frequently so it is constantly checking for new files to list rather then larger batches every 30 minutes?

Matt

Translate

nisaarAuthor

Gold ⭐️

@MattWho

Yes, its same issue if we run it frequently.

If I use Retry option on FetchSMB its able to fetch the file that failed during first attempt.

But, the concern is retry is impacting the scheduler, the next scheduler which is supposed to run after 30 minutes is not getting triggered, sometime it even skips 2 scheduler runs.

Is retry somehow impacting scheduler ?

my problem is not the error we get during FetchSMB, this can be resolved with retry option. But the concern is more on scheduler.

Thanks

Translate

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@nisaar

I expect that retry set on the Success relationship out of ListSMB is impacting your scheduling. I suspect that retry is blocking "successful" attempts by until both retry have been made which aligns with the skip twice you are seeing in scheduling. As I mentioned before, you should not be setting "retry" on any success relationships. This is an anti-pattern that will delay processing of any successful execution for the duration of the retry (Each retry would occur at the processors scheduled execution times).

Note: Scheduling of a NiFi component does not mean immediate execution. That execution depends on availability of threads to service the execution. NiFi has a "max timer driven thread count" configuration that establishes the thread pool from which all scheduled component threads come from.

So things like number of running components, number of concurrent tasks set on a given component, CPU insensitive components, etc can impact when a "scheduled" component is given a thread from the thread pool to execute.

I Thread pool does not impact scheduler unless the processor has been scheduled and never executes until within the 1.5 hours since it was initially scheduled (which I doubt in this case). The more logical cause because of the consistent behavior is the "retry" you have set on Success from ListSMB.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.