NiFi stuck data in queue between processors

@asand3r 

Here are my observations from  what you have shared:

1. You appear to be having a load balancing issue.  The LB icon on the connection indicates it is trying to actively load balance FlowFiles in that connection.   If load balancing is complete the  icon will look like this:  .  The  shared queue counts show that 1 node has reached queue threshold which would prevent it from receiving any more FlowFiles which includes from other nodes.  So I assume that the ~4,600 FlowFiles on first 2 nodes are destined for that third node but can not be sent because of queue threshold.  
2. Considering the observation above, i would focus you attention on the node with the queue threshold. Maybe disconnect it from cluster via the cluster UI and inspect the flow on that node directly.  Check the logs on that third node for any reported error or warn issues.
3. Perhaps EvaluateJson processor only on node 3 is having issues?  Maybe connectivity between first two nodes and and third node is having issues. Node 3 can't distribute any FlowFiles to nodes 1 or 2.  And nodes 1 and  2 can't distribute to node 3.  Maybe some sync issue happened node 3 for some reason has processor stopped. This may explain the stop and start of canvas getting things moving again.  If you just disconnect node 3 (one with queue threshold exceeded) only and then reconnect it back to cluster, do FlowFiles start moving?  At reconnection, node 3 will compare its local flow with cluster flow.  If you remove the LB connection configuration, do FlowFiles get processed?
4. I am curious why in this flow design you have setup a LB connection after the ConsumeKafka processor.  This processor creates a consumer group and should be configured according the number of partitions on the target topic to maximize throughput and preventing rebalancing. Let's say your topic has 15 partitions for example. Your consumeKafka processor would then be configured with 5 concurrent tasks.  3 nodes X 5 tasks = 15 consumers in the consumer group.  each consumer is assigned to a partition.   This spread the consumption across all your nodes removing the need for the load balanced connection configuration.
5. You are using a rather old version of Apache NiFi (~4 years old).  I'd encourage you to upgrade to take advantage of many bug, improvements, and security fixes.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt