Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Apache nifi cluster fail with around 11 millions rows

avatar
Explorer
Hello, in K8S I have deployed Nifi with 2 nodes and Request : 2 cpu 4Gi RAM / Limits : 8 cpu 16Gi RAM. Heap init is 2G and max 4.
My pipeline works with data requested from postgre database less than 10 millions rows (one table).
When I have more data (e.g 11 millions rows) that I want to get with executesql record my pipeline fail with this log :
2023-02-23 08:47:48,466 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
And next log is :
Caused by: java.nio.file.FileAlreadyExistsException: /opt/nifi/nifi-current/provenance_repository/lucene-8-index-1676991780200/_48.cfs 
What can be the reasons ? It's like RAM/CPU are not enough but not sure. 11 millions doesn't seem to be alot.
0 REPLIES 0