Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HandleHttpRequest 503 due to org.eclipse.jetty.io.EofException

avatar
Explorer

I have a workflow that receives data via HandleHttpRequest -> validates record -> HandleHttpResponse. We get about 3k data points per 5 minutes, but concentrated on the first 1-2 minutes.

 

I am getting these errors and having trouble finding how to solve it (or even if it is Nifi the issue).

HandleHttpRequest

  • Concurrent tasks 10
  • Container queue size 20000 (was at 10000)

HandleHttpResponse 

  • Concurrent tasks 10

StandardHttpContextMap

  • Maximum Outstanding Requests: 10000
  • Request Expiration: 1 min (tried 30s, 1.5min, 2min)

nifi.web.jetty.threads=200

 

I'm not a network engineer so this is a bit outside my knowledge, but any fingers pointing to a direction I can investigate would help greatly. Below are samples of some of the logs/bulletins.

 

2020-12-09 08:37:27,803 WARN [StandardHttpContextMap-3fbd160b-bc0f-3034-7cbc-54e701739667] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=3fbd160b-bc0f-3034-7cbc-54e701739667] Request from 10.0.2.2 timed out; responding with SERVICE_UNAVAILABLE
2020-12-09 08:37:27,803 ERROR [StandardHttpContextMap-3fbd160b-bc0f-3034-7cbc-54e701739667] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=3fbd160b-bc0f-3034-7cbc-54e701739667] Failed to respond with SERVICE_UNAVAILABLE message: java.lang.IllegalStateException: ABORTED

 

2020-12-09 08:39:01,077 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.HandleHttpResponse HandleHttpResponse[id=8c275c27-cb8e-3769-a2c5-08922f5ccfc2] Failed to respond to HTTP request for StandardFlowFileRecord[uuid=0e573cb2-3257-4935-985b-d83cefbc5bb6,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1607495941009-29472, container=default, section=800], offset=838276, length=3274],offset=0,name=b57af8ca-cec4-4181-992e-105e0541a980,size=3274] due to org.eclipse.jetty.io.EofException: org.eclipse.jetty.io.EofException
org.eclipse.jetty.io.EofException: null

 

2020-12-09 08:37:29,259 WARN [qtp292756264-19630] o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=07d42765-9481-357b-fd02-c3d58f00206c] Request from 10.0.2.2 cannot be processed, container queue is full; responding with SERVICE_UNAVAILABLE

 

 

1 REPLY 1

avatar
Explorer

Consequent to this we found that some of our devices has a newline character to the url they sent, e.g. instead of posting to

 

http://my.url

 

they would post to

 

http://my.url\n

 

When using Wireshark we could see http 404 errors with http://my.url%0A

 

The \n came from a mistake and removing it has stopped all the errors.

 

I'm interested to know if anyone has any insight into why this causes and error and how to mitigate this if possible (just as a fail-safe).