HandleHttpRequest 503 due to org.eclipse.jetty.io.EofException

Tokolosk — Wed, 09 Dec 2020 07:54:54 GMT

I have a workflow that receives data via HandleHttpRequest -> validates record -> HandleHttpResponse. We get about 3k data points per 5 minutes, but concentrated on the first 1-2 minutes.

I am getting these errors and having trouble finding how to solve it (or even if it is Nifi the issue).

HandleHttpRequest

Concurrent tasks 10
Container queue size 20000 (was at 10000)

HandleHttpResponse

Concurrent tasks 10

StandardHttpContextMap

Maximum Outstanding Requests: 10000
Request Expiration: 1 min (tried 30s, 1.5min, 2min)

nifi.web.jetty.threads=200

I'm not a network engineer so this is a bit outside my knowledge, but any fingers pointing to a direction I can investigate would help greatly. Below are samples of some of the logs/bulletins.

2020-12-09 08:37:27,803 WARN [StandardHttpContextMap-3fbd160b-bc0f-3034-7cbc-54e701739667] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=3fbd160b-bc0f-3034-7cbc-54e701739667] Request from 10.0.2.2 timed out; responding with SERVICE_UNAVAILABLE
2020-12-09 08:37:27,803 ERROR [StandardHttpContextMap-3fbd160b-bc0f-3034-7cbc-54e701739667] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=3fbd160b-bc0f-3034-7cbc-54e701739667] Failed to respond with SERVICE_UNAVAILABLE message: java.lang.IllegalStateException: ABORTED

2020-12-09 08:39:01,077 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.HandleHttpResponse HandleHttpResponse[id=8c275c27-cb8e-3769-a2c5-08922f5ccfc2] Failed to respond to HTTP request for StandardFlowFileRecord[uuid=0e573cb2-3257-4935-985b-d83cefbc5bb6,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1607495941009-29472, container=default, section=800], offset=838276, length=3274],offset=0,name=b57af8ca-cec4-4181-992e-105e0541a980,size=3274] due to org.eclipse.jetty.io.EofException: org.eclipse.jetty.io.EofException
org.eclipse.jetty.io.EofException: null

2020-12-09 08:37:29,259 WARN [qtp292756264-19630] o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=07d42765-9481-357b-fd02-c3d58f00206c] Request from 10.0.2.2 cannot be processed, container queue is full; responding with SERVICE_UNAVAILABLE

Re: HandleHttpRequest 503 due to org.eclipse.jetty.io.EofException

Tokolosk — Fri, 11 Dec 2020 09:55:23 GMT

Consequent to this we found that some of our devices has a newline character to the url they sent, e.g. instead of posting to

http://my.url

they would post to

http://my.url\n

When using Wireshark we could see http 404 errors with http://my.url%0A

The \n came from a mistake and removing it has stopped all the errors.

I'm interested to know if anyone has any insight into why this causes and error and how to mitigate this if possible (just as a fail-safe).

question Re: HandleHttpRequest 503 due to org.eclipse.jetty.io.EofException in Support Questions

HandleHttpRequest 503 due to org.eclipse.jetty.io.EofException

Re: HandleHttpRequest 503 due to org.eclipse.jetty.io.EofException