Failed to index Provenance Events org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed || Caused by: java.nio.file.FileSystemException

[#1]

Nifi provenance data is not visible and remains at 80% in searching for provenance events after upgrading from 1.13.0 to 1.15.3

Below are the error logs :

2022-04-05 23:51:22,905 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /provenance_repo/provenance_repository/lucene-8-index-1647749380623/_vd_Lucene80_0.dvd: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:410)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:406)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:254)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.<init>(Lucene80DocValuesConsumer.java:79)
at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsConsumer(Lucene80DocValuesFormat.java:161)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109)
at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108)
at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729)
at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.commit(LuceneEventIndexWriter.java:101)
at org.apache.nifi.provenance.index.lucene.EventIndexTask.commit(EventIndexTask.java:253)
at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:232)
... 6 common frames omitted

@MattWho Please help on this 

[#2 · Accepted solution]

@Neil_1992 
I agree that the first step here is to increase the open file limit for the user that owns your NiFi process.
check your current ulimit by becoming the user the user that owns your NiFi process and executing the "ulimit -a" command.

You can also inspect the /etc/security/limits.comf file.

NiFi can open a very large number of open files. The more FlowFile load, the larger the dataflows, the more concurrent tasks, etc all contribute to open file handles.

I recommend setting the ulimit to a very large value like 999999, restarting NiFi and seeing if your issue persists.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

[#3]

Hello @Neil_1992 

Why have you replace only these 4 nars ?

Have you got some logs ?

Thanks 

[#4]

I saw the issues were originating processors related to those nars , i back tracked the processor ids from the logs 

Caused by: java.lang.IllegalStateException: failed to create a child event loop
at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
at org.python.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:211)
... 126 common frames omitted
Caused by: org.python.netty.channel.ChannelException: failed to open a new selector
at org.python.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175)
at org.python.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:149)
at org.python.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127)
at org.python.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36)
at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
... 137 common frames omitted
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:130)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:69)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at org.python.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173)
... 141 common frames omitted

[#5]

Hello @Neil_1992 

Can you post the original error in attachment and where I saw the nars issues. I'm trying to find if i got the same issue.

Regards

[#6]

You are right @maykiwogno , the issue was resolved in one cluster but coming again in another cluster again and again , PFB logs for that

at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,600 ERROR [Timer-Driven Process Thread-22] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store
java.io.FileNotFoundException: /provenance_repo/provenance_repository/toc/5392629.toc (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190)
at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186)
at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:871)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:443)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:414)
at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:367)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,600 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,600 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,600 ERROR [Timer-Driven Process Thread-22] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store
java.io.FileNotFoundException: /provenance_repo/provenance_repository/toc/5392629.toc (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190)
at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186)
at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:871)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:443)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:414)
at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:367)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 ERROR [Timer-Driven Process Thread-22] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store
java.io.FileNotFoundException: /provenance_repo/provenance_repository/toc/5392629.toc (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190)
at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186)
at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:871)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:443)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:414)
at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:367)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 ERROR [Timer-Driven Process Thread-22] o.a.n.p.WriteAheadProvenanceRepository Failed to write events to the Event Store
java.io.FileNotFoundException: /provenance_repo/provenance_repository/toc/5392629.toc (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.nifi.provenance.toc.StandardTocWriter.<init>(StandardTocWriter.java:63)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.lambda$initialize$0(WriteAheadProvenanceRepository.java:125)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.tryRollover(WriteAheadStorePartition.java:273)
at org.apache.nifi.provenance.store.WriteAheadStorePartition.addEvents(WriteAheadStorePartition.java:190)
at org.apache.nifi.provenance.store.PartitionedEventStore.addEvents(PartitionedEventStore.java:104)
at org.apache.nifi.provenance.WriteAheadProvenanceRepository.registerEvents(WriteAheadProvenanceRepository.java:186)
at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:871)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:443)
at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:414)
at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:367)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)
2022-04-20 05:22:19,601 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Too many open files (Accept failed)
java.net.SocketException: Too many open files (Accept failed)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:107)
at java.lang.Thread.run(Thread.java:748)

[#7 · Accepted solution]

Hi All,

we need to increase open file limit for nifi as a service not user , so add the below line in your

nifi.service file and restart the server , it should resolve this issue

LimitNOFILE=999999

[#8]

Sorry @Neil_1992 where could I find nifi.service file, to add this property ?