Member since
05-19-2020
4
Posts
1
Kudos Received
0
Solutions
06-06-2020
02:48 PM
It has the same amount of threads and it does not hang. I did notice however that there were a number of timeouts, is it possible that InvokeHTTP is not timing out as intended? (the InvokeHTTP block was left using the default timeouts). There seems to be something more to this when a different process block with identical load and similar functionality does not exhibit this behaviour. Limiting the InvokeHTTP block to a single thread with a lower execution rate also exhibits the problem.
... View more
06-03-2020
09:23 AM
1 Kudo
Unfortunately the issue is still happening even after drastically reducing the scheduled execution time. What is strange is that replacing this block with an ExecuteStreamCommand block (which calls a Python function that does this same thing) does not seem to exhibit the same problem. Can you think of any reason as to why this is the case? @stevenmatison
... View more
05-19-2020
08:40 AM
These are the scheduling settings: NiFi 1.9.2 2 cores 4GB RAM [available to docker container running NiFi] 8 timer driver threads 5 event driven threads java.arg.2=-Xms1024m java.arg.3=-Xmx1024m
... View more
05-19-2020
03:37 AM
I have a NiFi flow [on a 2 processor machine using 8/5 thread counts] where the InvokeHTTP block is no longer processing files from the queue [shown below]: As you can see there are 8 processes running but these seem to be stuck for hours and the only way out of it is manually stopping and terminating the block. This results in the following log message: 2020-05-18 08:31:06,228 ERROR [Timer-Driven Process Thread-12 <Terminated Task>] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=f2f13d29-016b-1000-293d-c0cb04835881] Routing to Failure due to exception: java.net.SocketTimeoutException: timeout: java.net.SocketTimeoutException: timeout
java.net.SocketTimeoutException: timeout
at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.java:593)
at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.java:601)
at okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:146)
at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:791)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Before stopping the process, I had a look into the nifi-bootstrap.log file and noticed a number of similar messages. I'm not sure whether these are related since no direct reference to InvokeHTTP is made. "NiFi Web Server-556239" Id=556239 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2f7a83f7
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)
"OkHttp ConnectionPool" Id=213155 TIMED_WAITING on okhttp3.ConnectionPool@72d6189
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@7b6a1485
"Okio Watchdog" Id=556238 TIMED_WAITING on java.lang.Class@2b4512c0
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:361)
at okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:312) Any help/insights on this would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi