Member since
06-15-2016
7
Posts
2
Kudos Received
0
Solutions
09-15-2016
12:46 PM
I made some additional tests. The Problem wasn't Nifi's invokeHTTP. For the test I created a simple php-Page, which sleeps for n seconds. <?php
echo date('h:i:s') . "\n";
sleep(420);
echo date('h:i:s') . "\n";
?>
When I send a request directly to the Page, it works (some tests run for more than an hour). The problem seems to be the loadbalancer, wich throws an "Connection reset by peer" after a while.
... View more
09-14-2016
01:32 PM
1 Kudo
Hi @Pierre Villard Thank you for the answer. I set the timeout to 0s, but it still throw the same exception. I think, it's not a problem of the third party, because otherwhise I will receive an 50x-http exception. Their developer says, they get all the records and process it successfully. But I just get the exception in the error in the log (including the yellow bumble in the ui). The connection reset apperas after +/- 6 min.
... View more
09-09-2016
07:32 AM
1 Kudo
I have to send some data with InvokeHttp to a third party application. Because it's a long running job, i have set the connection timeout to 9000 secs. The problem I have is, that nifi throws after a while a connection reset exception. 2016-09-09 08:11:24,123 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.InvokeHTTP
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196) ~[na:1.7.0_67]
at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[na:1.7.0_67]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) ~[na:1.7.0_67]
at sun.security.ssl.InputRecord.read(InputRecord.java:480) ~[na:1.7.0_67]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) ~[na:1.7.0_67]
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) ~[na:1.7.0_67]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) ~[na:1.7.0_67]
at okio.Okio$2.read(Okio.java:139) ~[okio-1.6.0.jar:na]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:211) ~[okio-1.6.0.jar:na]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306) ~[okio-1.6.0.jar:na]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300) ~[okio-1.6.0.jar:na]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196) ~[okio-1.6.0.jar:na]
at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:739) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.internal.http.HttpEngine.access$200(HttpEngine.java:87) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:724) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:578) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.Call.getResponse(Call.java:287) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205) ~[okhttp-2.7.1.jar:na]
at com.squareup.okhttp.Call.execute(Call.java:80) ~[okhttp-2.7.1.jar:na]
at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:568) ~[nifi-standard-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059) [nifi-framework-core-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_67]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_67]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_67]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
I think that nifi has an internal max timeout. Is it possible to increase this?
... View more
Labels:
- Labels:
-
Apache NiFi
09-08-2016
11:14 AM
Hi, I had the same Exception. I solved the problem by creating the User ANONYMOUS on the kafka broker nodes.
... View more
06-20-2016
01:22 PM
I received the same date (perhaps I did something wrong 🙂 ). I solved the problem with the following UpdateAttribute: ${date:toDate("yyyy-MM-dd"):toNumber():plus(86400000):format("yyyy-MM-dd"):append(${date:toDate("yyyy-MM-dd HH:mm:ss"):toNumber():format(" HH:mm:ss")})}
It works, but I'm still not very happy with this solution.
... View more
06-16-2016
09:13 AM
I need next day as attribute. As I know, date manipulation by adding ms (e.g. ${now():toNumber():plus(86400000) ). With this solution I have a problem with the daylight saving: If date = "2016-10-29 12:00:00" ${date:toDate("yyyy-MM-dd HH:mm:ss"):toNumber():plus(86400000):format("yyyy-MM-dd HH:mm:ss")} This code results: "2016-10-30 11:00:00" I have built a pyhton solution, but I'm not realy happy with it: import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
from datetime import datetime
from datetime import timedelta
flowFile = session.get()
if (flowFile != None):
inDate = flowFile.getAttribute('inputDate')
outputDateName = flowFile.getAttribute('output date attribute name')
unit = flowFile.getAttribute('python timedelta unit')
quantity = int(flowFile.getAttribute('quantity'))
pythonDateFormat = flowFile.getAttribute('python date format')
kwargs = {unit: quantity }
outputDt = (datetime.strptime(inDate, pythonDateFormat) + timedelta(**kwargs)).strftime(pythonDateFormat)
flowFile = session.putAttribute(flowFile, outputDateName, outputDt)
session.transfer(flowFile, REL_SUCCESS)
Exist another solution with the given Nifi Processors? I'm using HDF 1.2.0.0.
... View more
Labels:
- Labels:
-
Apache NiFi