Hello @MattWho thanks for the information.  I'm already running the process only on the Primary node.  I will monitor and take a thread dump if it occurs again.  ... View more

@vafs I'm using version 1.23.2.  I'm already using Tracking Timestamps within ListS3.  ... View more

Hello Matt, The files were just sitting in the S3 bucket.  It appears that ListS3 process just didn't response.  After checking the ListS3 state no entry listed for the files at that time.  I had to stop the ListS3 process then start it before it pickup the files.   No error indicated that there as an issue with ListS3, no alert within the log for this ListS3 processor.  What is the exact version of Apache NiFi being used? 1.23.2 Single NiFi instance or a multi-node NiFi cluster setup? 3 Nodes Clusters How many "running" processors on your canvas? 2103 How large is the NiFi Max Timer Driven Thread pool set to (default is 10, but typically this is set to 2 to 4 times the number of cores on the NiFi host). 10 Monitoring of CPU load average with your flow running will allow you to determine if you can increase this even more. Okay f you saw a small number displayed on the processor indicating it was scheduled to execute while it as not producing any FlowFiles, you could take a serious of thread dumps which you could inspect to see if the listS3 processor thread was making any progress or just blocked/waiting. Not seeing this at the moment. Regards ... View more

Hello Matt, Thanks for the information.  With the flow.json.gz if a process group get deleted then the entire fflow.json.gz need to be restore.  I was wondering if there is another way to backup the running progress groups without taking a complete template backup that I can backup that will list the individual process groups so I can choose the one that's require a restore instead of restore all the process groups.  Thanks! ... View more

Hello Matt, Its configured via UDP, I understood that there is no way to track data loss. Is there anything I can check from Nifi that may contribute to this issue ?  For example if systems resource are high or other factor.  Thanks! ... View more

Hello @MattWho @SAMSAL @steven-matison @DigitalPlumber @cotopaul , after confirming the access I'm able to verified I can access the bucket from aws cli on the same system Nifi is running on and download files.  The problem I'm having now is that the ListS3 pickup the list of the files from the bucket but the FetchS3 doesn't do anything.  When I enabled debugging on the FetchS3 I receiving the following error message.    FetchS3 Error message    FetchS3 Configuration ListS3 Configuration  Any suggestion to what is causing the issue?   ... View more