NiFi - Flow file stuck in queue

[#1]

In Apache NiFi, the flow file got stuck permanently in queue when the downstream processor has some errors.

I am looking to route the flow file to failure relationship so that I can handle the failure and extract the failure log. But in some cases the flow file got stuck in queue permanently which makes routing to failure relationship impossible.

Is there a way to resolve the issue of flow file getting stuck? If not, is there a way to get notified that a particular flow file waits in which queue and which downstream processor in the processor group has error, so that I can track the failure?

Downstream processor's settings and configuration:

Note: Nifi is the part of the application we use to process files. We are looking for handling failure in Nifi processors without explicitly visiting NiFi UI every time (Getting to know about failures in NiFi processors from the system instead of visiting NiFi UI ). In the case mentioned above, it is impossible to route to failure relationship.

[#2]

FWIW - I did set up a monitoring mechanism for bulletins.  I have not really made a way to automatically associate the bulletins back to the flowfiles, but I have utilized the cluster-level Controller Services for this.

1. Under Reporting Task Controller Services:
   1. DBCPConnectionPool for the database connection
   2. DatabaseRecordSink pointed at a table matching the internal BULLETINS data structure
2. Under Reporting Tasks:
   1. QueryNiFiReportingTask which pulls from BULLETINS and structures into a record matching the table used by the above DatabaseRecordSink.

The reporting task is set to run every 5 minutes, so that it captures each set of bulletins before it cycles out of the in-memory data structures.

I then have other components reporting on records in that table for tracking by our other tools.  (And the flowfile ID is part of the bulletins structure - so it might be linkable back to whatever is stuck in the upstream connection.

[#3]

Hi Kellerj. Thank you for the reply. I will try this approach for my case. Can you please send the sample template for the flow you mentioned?

[#4]

Sure.  I had to make some extract it from a larger PG and make some adjustments due to key code being in a parameter.  But the remainder of the parameters are really placeholders for your environment information.  The main complication might be that I used a few external controller services that you will have to replace.  But their names in the externalControllerServices JSON should help you replace them.