You are right @maykiwogno , the issue was resolved in one cluster but coming again in another cluster again and again , PFB logs for that   at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,600 ERROR [Timer-Driven Process Thread-22] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store java.io.FileNotFoundException: /provenance_repo/provenance_repository/toc/5392629.toc (Too many open files) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63) at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125) at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273) at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190) at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104) at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186) at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:871) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:443) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:414) at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:367) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,600 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,600 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,600 ERROR [Timer-Driven Process Thread-22] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store java.io.FileNotFoundException: /provenance_repo/provenance_repository/toc/5392629.toc (Too many open files) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63) at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125) at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273) at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190) at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104) at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186) at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:871) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:443) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:414) at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:367) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 ERROR [Timer-Driven Process Thread-22] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store java.io.FileNotFoundException: /provenance_repo/provenance_repository/toc/5392629.toc (Too many open files) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63) at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125) at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273) at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190) at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104) at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186) at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:871) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:443) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:414) at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:367) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 ERROR [Timer-Driven Process Thread-22] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store java.io.FileNotFoundException: /provenance_repo/provenance_repository/toc/5392629.toc (Too many open files) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63) at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125) at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273) at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190) at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104) at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186) at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:871) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:443) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:414) at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:367) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748) 2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed) java.net.SocketException: Too many open files (Accept failed) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107) at java.lang.Thread.run(Thread.java:748)   ... View more

I saw the issues were originating processors related to those nars , i back tracked the processor ids from the logs    Caused by: java.lang.IllegalStateException: failed to create a child event loop at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) at org.python.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77) at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72) at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:211) ... 126 common frames omitted Caused by: org.python.netty.channel.ChannelException: failed to open a new selector at org.python.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175) at org.python.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:149) at org.python.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127) at org.python.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36) at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ... 137 common frames omitted Caused by: java.io.IOException: Too many open files at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:130) at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:69) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) at org.python.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173) ... 141 common frames omitted ... View more

I have found the root cause to be some of the nars of 1.15.3 version , When i removed those and replaced with same nars from 1.13.0 , the issue did not appear , below is the list of nars i removed later replaced with 1.13.0 version sudo mv /apps/nifi-1.15.3/lib/nifi-distributed-cache-services-nar-1.15.3.nar /apps sudo mv /apps/nifi-1.15.3/lib/nifi-hadoop-nar-1.15.3.nar /apps/ sudo mv /apps/nifi-1.15.3/lib/nifi-standard-nar-1.15.3.nar /apps sudo mv /apps/nifi-1.15.3/lib/nifi-scripting-nar-1.15.3.nar /apps/ ... View more

Thanks for your valuable inputs ... View more

Adding below to /etc/security/limits.conf and Restart the nodes twice with adding nodes one after another syncing with primary resolved the Issue   * hard nofile 500000 * soft nofile 500000 root hard nofile 500000 root soft nofile 500000 nifi hard nofile 500000 nifi soft nofile 500000 ... View more

@MattWho , can you please help us on this , looks like this is common with Nifi 1.15.3 version which happens to be the latest one ... View more

Hi @MattWho ,   We are struggling to connect to MS Access DB from Nifi 1.13.0 using Ucanaccess 5.0.1 Driver , we are getting error as "Given file does not exist"  Can you please suggest on this or if you have used any other driver  or any information around that   There is a Cdata driver which works but that is a licensed product , we are looking for something in the open source region ... View more

I am able to see some anomalies in the flow.xml.gz file when i unzipped and looking around it , one of the property value inside of it is having almost a million null values , this is must be some kind of bug and resulting in this bytearray outofmemory error , take a look at the below snap     ... View more