We have a nifi processor that collects data with httphandlerequest. Recently we installed tshark to see if all the posts we are receiving are being captured by Nifi:
In the below graph we are plotting the flowfiles coming out of the httphandlerequest processor (dark blue) and also plotting a count of the captured posts from tshark every 5 minutes.
There are events where we seem to be picking up more posts in tshark than in Nifi (highlighted in green below).
This is how our processor is set up:
I have tried playing with settings to see if it changes the behaviour and I do not know if I'm either still too low below the threshold or if these are the wrong settings to change.
The settings I changed are both the thread counts below and the concurrent tasks for the processor.
We expect about 5000 datapoints over 2 minutes, currently scaled down to about 250 per 5 minutes - which seems like Nifi should be able to handle easily.
Are we looking in the wrong place?
Edit: just to add that the process is handlehttprequest -> validateRecord -> handlehttpresponse
