Hi,
This week I started facing an issue with a PutFile processor that was running for over a year without problems.
This processor is getting blocked and start enqueuing flowfiles. When I tried to terminate the process inside Nifi it ends; but no matter how many times i try to restart it, the processor never works again.
My next step is to shutdown Nifi. The shutdown process ends OK, but the Windows process never ends and kept temporary files locked and RAM memory allocate.
But the worse part is that this process is impossible to kill. I tried to kill it from the task manager and got access error. Also tried to kill it from cmd (as administrator) using taskkill and wimc commands but the process does not died.
Until now, the only solution that I have found is to restart the server. But even this solution works half way, because if I send the reboot from the server; this one stays (I guess) trying to kill the process and doesn't shut down. The only thing that has worked is to send the reboot from the administration console of my hosting provider.
The following is the nifi-dump log:
"Timer-Driven Process Thread-1 <Terminated Task>" Id=61 RUNNABLE (in native code)
at java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.CreateFile0(Native Method)
at java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:71)
at java.base@11.0.15/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:308)
at java.base@11.0.15/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
at java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(WindowsFileSystemProvider.java:330)
at java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkAccess(WindowsFileSystemProvider.java:367)
at java.base@11.0.15/sun.nio.fs.AbstractFileSystemProvider.exists(AbstractFileSystemProvider.java:151)
at java.base@11.0.15/java.nio.file.Files.exists(Files.java:2435)
at org.apache.nifi.processors.standard.PutFile.onTrigger(PutFile.java:315)
at app//org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base@11.0.15/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base@11.0.15/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
Number of Locked Synchronizers: 1
"Timer-Driven Process Thread-10" Id=84 RUNNABLE (in native code)
at java.base@11.0.15/java.io.WinNTFileSystem.list(Native Method)
at java.base@11.0.15/java.io.File.normalizedList(File.java:1165)
at java.base@11.0.15/java.io.File.listFiles(File.java:1258)
at org.apache.nifi.processors.standard.GetFile.performListing(GetFile.java:321)
at org.apache.nifi.processors.standard.GetFile.onTrigger(GetFile.java:386)
at app//org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base@11.0.15/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base@11.0.15/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
Number of Locked Synchronizers: 2
- java.util.concurrent.locks.ReentrantLock$NonfairSync@6d7d76f3
- java.util.concurrent.ThreadPoolExecutor$Worker@153e2f2f
"Timer-Driven Process Thread-11" Id=6746 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bb8fee2
at java.base@11.0.15/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.15/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11.0.15/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
"Timer-Driven Process Thread-12 <Terminated Task>" Id=6750 RUNNABLE (in native code)
at java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.CreateFile0(Native Method)
at java.base@11.0.15/sun.nio.fs.WindowsNativeDispatcher.CreateFile(WindowsNativeDispatcher.java:71)
at java.base@11.0.15/sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:308)
at java.base@11.0.15/sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:168)
at java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkReadAccess(WindowsFileSystemProvider.java:330)
at java.base@11.0.15/sun.nio.fs.WindowsFileSystemProvider.checkAccess(WindowsFileSystemProvider.java:367)
at java.base@11.0.15/sun.nio.fs.AbstractFileSystemProvider.exists(AbstractFileSystemProvider.java:151)
at java.base@11.0.15/java.nio.file.Files.exists(Files.java:2435)
at org.apache.nifi.processors.standard.PutFile.onTrigger(PutFile.java:315)
at app//org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent$$Lambda$1526/0x0000000800f54040.run(Unknown Source)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base@11.0.15/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base@11.0.15/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@bbdcc1d
"Timer-Driven Process Thread-13" Id=6751 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bb8fee2
at java.base@11.0.15/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.15/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11.0.15/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
at java.base@11.0.15/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11.0.15/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11.0.15/java.lang.Thread.run(Thread.java:829)
In the Nifi-app.log does not exists a related message.
As I mentioned before, this processor (or its process) has no had changes recently. But a new process that has several PutFile processor with high usage was recently deployed on the Nifi instance. However this new process is not having issues and is working without problem while the old process is having the described problem.
I'll appreciate your help
--------------------------
Nifi 1.14.0
OpenJDK 64-Bit Server VM Microsoft-32930 (build 11.0.15+10-LTS, mixed mode)
Windows Server Std 2016 (10.0.14.393 b 14393)
Allocate RAM 8G
Announcements
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
