In Apache NIFI requests are queued and showing socket timeout exception ,not getting processed

[#1]

1000 requests are queued in NIFI , invoke HTTP processor is showing socket timeout exception , i have added a self loop in invoke http processor so that it keep on retrying . These requests are stuck permanently.

My issue is endpoint which invoke http processor is trying to invoke is up and running .  i checked from postman i am able to get success response.

2023-11-08 12:17:34,783 ERROR [Timer-Driven Process Thread-138] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=b48ff3b8-befb-32c2-b670-83445e1873f2] Routing to Failure due to exception: java.net.SocketTimeoutException: connect timed out: java.net.SocketTimeoutException: connect timed out
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394)
at java.net.Socket.connect(Socket.java:606)
at okhttp3.internal.platform.Platform.connectSocket(Platform.java:124)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:223)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:149)
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:195)
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)

[#2 · Accepted solution]

@Rohit1997jio  If your endpoint test works with Postman, and your invokeHttp is setup similary, the above error suggest the nifi node cannot connect to the endpoint.  You would need to ensure you have network connectivity from nifi host to endpoint.   

[#3 · Accepted solution]

i created a new flow with same end points , some requests are getting successful , but still few requests are still getting stuck in invoke http processor giving socket timeout exception

[#4 · Accepted solution]

@Rohit1997jio  If your endpoint test works with Postman, and your invokeHttp is setup similary, the above error suggest the nifi node cannot connect to the endpoint.  You would need to ensure you have network connectivity from nifi host to endpoint.   

[#5 · Accepted solution]

i created a new flow with same end points , some requests are getting successful , but still few requests are still getting stuck in invoke http processor giving socket timeout exception

[#6 · Accepted solution]

Sometimes I need to increase the timeouts in the processor configurations.    You may also need to reduce the execution time for the processor incase the endpoint cannot handle too many requests at once.

[#7]

Did you find any solution about this issue ? I have exactly the same problem.

An invoke HTTP get a time out on the flow file. A retry is implemented and the flow file get stucked with a timeout each time.

Works fine with same configuration to call API with postman and a curl from server nifi node.

The only way is to restart the invoke http processor and after few hours and some flow files a new occurence appears.

Error :

2024-07-17 14:55:06,783 ERROR [Timer-Driven Process Thread-17] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=ba9d2024-8df1-323d-b0ba-54402272b188] Routing to Failure due to exception: java.net.SocketTimeoutException: timeout: java.net.SocketTimeoutException: timeout
java.net.SocketTimeoutException: timeout
at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.java:593)
at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.java:601)
at okhttp3.internal.http2.Http2Stream$FramingSink.emitFrame(Http2Stream.java:510)
at okhttp3.internal.http2.Http2Stream$FramingSink.close(Http2Stream.java:538)
at okio.RealBufferedSink.close(RealBufferedSink.java:236)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:63)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:850)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1174)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

Thanks