Nifi InvokeHttp Connection Reset

[#1]

I have to send some data with InvokeHttp to a third party application.

Because it's a long running job, i have set the connection timeout to 9000 secs.

The problem I have is, that nifi throws after a while a connection reset exception.

2016-09-09 08:11:24,123 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.InvokeHTTP
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:196) ~[na:1.7.0_67]
        at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[na:1.7.0_67]
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) ~[na:1.7.0_67]
        at sun.security.ssl.InputRecord.read(InputRecord.java:480) ~[na:1.7.0_67]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) ~[na:1.7.0_67]
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) ~[na:1.7.0_67]
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) ~[na:1.7.0_67]
        at okio.Okio$2.read(Okio.java:139) ~[okio-1.6.0.jar:na]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:211) ~[okio-1.6.0.jar:na]
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306) ~[okio-1.6.0.jar:na]
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300) ~[okio-1.6.0.jar:na]
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) ~[okio-1.6.0.jar:na]
        at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:739) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.internal.http.HttpEngine.access$200(HttpEngine.java:87) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:724) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:578) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.Call.getResponse(Call.java:287) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205) ~[okhttp-2.7.1.jar:na]
        at com.squareup.okhttp.Call.execute(Call.java:80) ~[okhttp-2.7.1.jar:na]
        at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:568) ~[nifi-standard-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059) [nifi-framework-core-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_67]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_67]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_67]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]

I think that nifi has an internal max timeout. Is it possible to increase this?

[#2]

Hi @Marius Müller,

Could you try setting connection timeout and read timeout to 0s to completely remove the timeout?

Also, after how long do you have this exception? Could it be because of the third party application?

If it does not help, I'll to reproduce it on my side.

[#3]

Hi @Pierre Villard

Thank you for the answer.

I set the timeout to 0s, but it still throw the same exception.

I think, it's not a problem of the third party, because otherwhise I will receive an 50x-http exception. Their developer says, they get all the records and process it successfully.

But I just get the exception in the error in the log (including the yellow bumble in the ui).

The connection reset apperas after +/- 6 min.

[#4]

I made some additional tests.

The Problem wasn't Nifi's invokeHTTP.

For the test I created a simple php-Page, which sleeps for n seconds.

<?php
echo date('h:i:s') . "\n";
sleep(420);
echo date('h:i:s') . "\n";
?>

When I send a request directly to the Page, it works (some tests run for more than an hour).

The problem seems to be the loadbalancer, wich throws an "Connection reset by peer" after a while.

[#5]

OK, great, thanks for letting us know, I was about to make some tests!