Hello, in K8S I have deployed Nifi with 2 nodes and Request : 2 cpu 4Gi RAM / Limits : 8 cpu 16Gi RAM. Heap init is 2G and max 4.
My pipeline works with data requested from postgre database less than 10 millions rows (one table).
When I have more data (e.g 11 millions rows) that I want to get with executesql record my pipeline fail with this log :
2023-02-23 08:47:48,466 ERROR [Index Provenance Events-2] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
And next log is :
Caused by: java.nio.file.FileAlreadyExistsException: /opt/nifi/nifi-current/provenance_repository/lucene-8-index-1676991780200/_48.cfs
What can be the reasons ? It's like RAM/CPU are not enough but not sure. 11 millions doesn't seem to be alot.