Silver ⭐️⭐️⭐️

Question

ListS3 Not Picking Up Files

6 replies
1301 views
Translate All

Hello all,
I have an ListS3 process that access an S3 bucket to pickup files. This process running fine since deployed in October 2025. Today, noticed that the files didn't get pickup and they were no error reported on the ListS3. The state didn't list the current files in S3 waiting to pickup. I had to stop and start the process before it get pickup. I enabled debug to see if it will report any issue while trying to connect and also updated the run scheduled from 0 to 5 seconds.

Any other suggestion or recommendation on what I can do to be prepare to help identify the issue if it occurred again in the future?

Thanks!

Apache NiFi

vafs

Gold ⭐️⭐️⭐️

Hello @jame1997,

Can you tell us which version of NiFi are you using?
I want to confirm if there is any bug reported.

Something you can try is to change the State Manager to Tracking Timestamps which is more reliable.

Translate

Regards, Andrés Fallas -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs-up button.

jame1997Author

Silver ⭐️⭐️⭐️

@vafs I'm using version 1.23.2. I'm already using Tracking Timestamps within ListS3.

Translate

MattWho

Titanium ⭐️⭐️⭐️

@jame1997

There is not enough information yet to say what was experienced here.
When you say "had to stop and start the process", does this mean you had to stop and start NiFi, or stop and start only the ListS3 processor to get listing to start working again?
When the listS3 was not producing any FlowFiles, was it showing an small number in upper right corner indicating an active thread?

When this listS3 is not working, is the outbound connection from the processor "red" indicating backpressure is being applied preventing the processor from getting scheduled?

What is the exact version of Apache NiFi being used? Single NiFi instance or a multi-node NiFi cluster setup? How many "running" processors on your canvas? How large is the NiFi Max Timer Driven Thread pool set to (default is 10, but typically this is set to 2 to 4 times the number of cores on the NiFi host). Monitoring of CPU load average with your flow running will allow you to determine if you can increase this even more.

Perhaps the canvas was. thread starved. As more dataflows are built on the canvas, there is more chance the default thread pool may not be large enough to run your flow smoothly. Any long running threads can prevent other processors that are scheduled from getting a thread for extended periods of time. If you saw a small number displayed on the processor indicating it was scheduled to execute while it as not producing any FlowFiles, you could take a serious of thread dumps which you could inspect to see if the listS3 processor thread was making any progress or just blocked/waiting.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

jame1997Author

Silver ⭐️⭐️⭐️

Hello Matt,
The files were just sitting in the S3 bucket. It appears that ListS3 process just didn't response. After checking the ListS3 state no entry listed for the files at that time. I had to stop the ListS3 process then start it before it pickup the files. No error indicated that there as an issue with ListS3, no alert within the log for this ListS3 processor.

What is the exact version of Apache NiFi being used? 1.23.2

Single NiFi instance or a multi-node NiFi cluster setup? 3 Nodes Clusters

How many "running" processors on your canvas? 2103

How large is the NiFi Max Timer Driven Thread pool set to (default is 10, but typically this is set to 2 to 4 times the number of cores on the NiFi host). 10

Monitoring of CPU load average with your flow running will allow you to determine if you can increase this even more. Okay

f you saw a small number displayed on the processor indicating it was scheduled to execute while it as not producing any FlowFiles, you could take a serious of thread dumps which you could inspect to see if the listS3 processor thread was making any progress or just blocked/waiting. Not seeing this at the moment.

Regards

Translate

MattWho

Titanium ⭐️⭐️⭐️

@jame1997

Since you are running a NiFi multi-node cluster, your ListS3 processor should be configured so this it only scheduled on the Primary node. The outbound connection feeds a FetchS3Object processor. That connection should have the "Load Balance Strategy" configured for Round Robin (this allows 0 byte listed FlowFiles to be distributed across all your nodes so that each node share the workload of fetching the content and processing it). ListS3 should only have the "1" concurrent task set. FetchS3Object can have multiple concurrent task if needed to keep up with listing. Also keep in mind that using Tracking timestamps can result in objects being missed and not listed. "Tracking Entities" is a more robust option but requires a map cache to hold those entities metadata.

With 2,103 running processors and a timer driven thread pool of only 10, you may see delays in processors getting threads to do the work once they are scheduled for execution. What does not makes sense here is your statement that all you needed to do to get listS3 executing successfully was to stop just that processor and start it again. A common issue seen with concurrent tasks is users setting high concurrent tasks on some processor impacting other processors ability to get a thread.

Otherwise there is not enough info here to speculate on the cause. I looked through Apache NiFi jira project for any known bugs that may relate to what you described and found none unless other details are missing. I can only suggest capturing a serious of thread dumps (spaced apart by 5 mins) should issue occur again and analyze those to see what the listS3 thread might be doing.

Maybe take a look at these listS3 bugs that impact your version and are fixed in a newer release:

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

jame1997Author

Silver ⭐️⭐️⭐️

Hello @MattWho thanks for the information. I'm already running the process only on the Primary node. I will monitor and take a thread dump if it occurs again.

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.