I have a situation where occasionally flow files appear to be stuck in a queue before an InvokeHttp processor. The flow file attributes show that none of the files are Penalized. I'm not sure how to diagnose the issue, any pointers would be appreciated.
If you are forced to "terminate" the processor after requesting it to stop, this can mean that you have a stuck or very long running process thread. The "terminate" does not actually kill the thread, it disassociates that JVM thread from the processor and he current FlowFile that thread is associated with. The terminated thread continues to run until it completes (NiFi does request thread to quit/exit when terminating it, but success of that varies since not all process support that ability). Any terminated threads still active will be represented in the processor by a small number in parentheses (1) displayed in its upper right corner. The previously associated FlowFile is left on the inbound connection and will be handled based on queue priority when the consuming processor is started again. If the "terminated" thread should eventually complete, any output/return from that thread including logging is just sent to null.
To figure out what is going on when you have a seemingly hung thread is to get a series of NiFi thread dumps:
./nifi.sh dump <dump-filename-nodenum-01.txt>
Getting at least 3 dumps at an interval of 2-5 minutes apart is usually good.
What you are looking for is a thread associated with your processor class (invokeHTTP in this case) where the same thread number exists in every thread dump collected. Then you will want to luck that the thread stack to see if all are identical. If thread output changes between thread dump outputs, it indicates that thread is not hung but rather just long running. If thread dump output does not change, you'll want to dig in to that output to see what it is waiting on.