Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

HTTP Post count from tshark differs to httphandlerequest


We have a nifi processor that collects data with httphandlerequest. Recently we installed tshark to see if all the posts we are receiving are being captured by Nifi:


In the below graph we are plotting the flowfiles coming out of the httphandlerequest processor (dark blue) and also plotting a count of the captured posts from tshark every 5 minutes.


There are events where we seem to be picking up more posts in tshark than in Nifi (highlighted in green below).


This is how our processor is set up:



I have tried playing with settings to see if it changes the behaviour and I do not know if I'm either still too low below the threshold or if these are the wrong settings to change.

The settings I changed are both the thread counts below and the  concurrent tasks for the processor.


We expect about 5000 datapoints over 2 minutes,  currently scaled down to about 250 per 5 minutes - which seems like Nifi should be able to handle easily.


Are we looking in the wrong place?


Edit: just to add that the process is handlehttprequest -> validateRecord -> handlehttpresponse



Just to add to the above, I am getting a lot of the below errors. All the devices sending data via http POSTS are on cellular networks with some having bad signal. Could this be causing the difference between posts the HandleHttpRequest processor receives vs the amount of flowfiles it creates?


ERROR POST from HandleHttpRequest[id=da9f6288-5359-38e9-cb70-1c832b186f63] Failed to receive content from HTTP Request from due to java.util.concurrent.TimeoutException: Idle timeout expired: 60000/60000 ms: java.util.concurrent.TimeoutException: Idle timeout expired: 60000/60000 ms


Sorry about the bump but I'm really struggling to get to the bottom of this.