HandleHttpRequest 503 due to org.eclipse.jetty.io.EofException

[#1]

I have a workflow that receives data via HandleHttpRequest -> validates record -> HandleHttpResponse. We get about 3k data points per 5 minutes, but concentrated on the first 1-2 minutes.

I am getting these errors and having trouble finding how to solve it (or even if it is Nifi the issue).

HandleHttpRequest

• Concurrent tasks 10
• Container queue size 20000 (was at 10000)

HandleHttpResponse 

• Concurrent tasks 10

StandardHttpContextMap

• Maximum Outstanding Requests: 10000
• Request Expiration: 1 min (tried 30s, 1.5min, 2min)

nifi.web.jetty.threads=200

I'm not a network engineer so this is a bit outside my knowledge, but any fingers pointing to a direction I can investigate would help greatly. Below are samples of some of the logs/bulletins.

2020-12-09 08:37:27,803 WARN [StandardHttpContextMap-3fbd160b-bc0f-3034-7cbc-54e701739667] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=3fbd160b-bc0f-3034-7cbc-54e701739667] Request from 10.0.2.2 timed out; responding with SERVICE_UNAVAILABLE
2020-12-09 08:37:27,803 ERROR [StandardHttpContextMap-3fbd160b-bc0f-3034-7cbc-54e701739667] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=3fbd160b-bc0f-3034-7cbc-54e701739667] Failed to respond with SERVICE_UNAVAILABLE message: java.lang.IllegalStateException: ABORTED

2020-12-09 08:39:01,077 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.HandleHttpResponse HandleHttpResponse[id=8c275c27-cb8e-3769-a2c5-08922f5ccfc2] Failed to respond to HTTP request for StandardFlowFileRecord[uuid=0e573cb2-3257-4935-985b-d83cefbc5bb6,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1607495941009-29472, container=default, section=800], offset=838276, length=3274],offset=0,name=b57af8ca-cec4-4181-992e-105e0541a980,size=3274] due to org.eclipse.jetty.io.EofException: org.eclipse.jetty.io.EofException
org.eclipse.jetty.io.EofException: null

2020-12-09 08:37:29,259 WARN [qtp292756264-19630] o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=07d42765-9481-357b-fd02-c3d58f00206c] Request from 10.0.2.2 cannot be processed, container queue is full; responding with SERVICE_UNAVAILABLE

[#2]

Consequent to this we found that some of our devices has a newline character to the url they sent, e.g. instead of posting to

http://my.url

they would post to

http://my.url\n

When using Wireshark we could see http 404 errors with http://my.url%0A

The \n came from a mistake and removing it has stopped all the errors.

I'm interested to know if anyone has any insight into why this causes and error and how to mitigate this if possible (just as a fail-safe).