Hi Folks,   I've been facing an issue with nifi pod restarting only when persistence is enabled, Below is the app.log excerpt.   I doubt this could be related to provenance. Can someone help?   APP.log 2023-01-20 12:47:20,598 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loading Application Properties [/opt/nifi/nifi-current/./conf/nifi.properties] 2023-01-20 12:47:29,181 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2023-01-20 12:47:29,181 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 2 milliseconds 2023-01-20 12:47:29,192 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 checkpointed with 0 Records and 0 Swap Files in 11 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1 2023-01-20 12:47:29,991 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 117 properties from system properties and environment variables 2023-01-20 12:47:29,991 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 117 properties. Including precedence overrides effective accessible registry key size is 117 2023-01-20 12:47:31,323 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository 2023-01-20 12:47:31,328 INFO [main] o.apache.nifi.controller.FlowController Creating Provenance Repository [org.apache.nifi.provenance.WriteAheadProvenanceRepository] 2023-01-20 12:47:31,598 INFO [main] o.a.n.p.store.WriteAheadStorePartition After recovering ../provenance_repository, next Event ID to be generated will be 0 2023-01-20 12:47:31,754 INFO [main] o.a.n.p.index.lucene.LuceneEventIndex Will avoid re-indexing Provenance Events because the newest index is defunct, so it will be re-indexed in the background 2023-01-20 12:47:31,806 INFO [main] o.apache.nifi.controller.FlowController Creating Content Repository [org.apache.nifi.controller.repository.FileSystemRepository] 2023-01-20 12:47:31,827 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 9126805504 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2023-01-20 12:47:31,842 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2023-01-20 12:49:29,376 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 checkpointed with 0 Records and 0 Swap Files in 182 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1 2023-01-20 12:49:55,078 INFO [main] org.apache.nifi.NiFi Launching NiFi... 2023-01-20 12:49:55,188 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loading Application Properties [/opt/nifi/nifi-current/./conf/nifi.properties] 2023-01-20 12:49:55,198 INFO [main] org.apache.nifi.NiFi Application Properties loaded [142] 2023-01-20 12:49:55,291 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 40217 2023-01-20 12:49:55,314 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap 2023-01-20 12:49:55,390 INFO [main] org.apache.nifi.nar.NarUnpacker Expanding 122 NAR files with all processors...     Persistence settings configStorage: size: 100Mi authconfStorage: size: 100Mi # Storage capacity for the 'data' directory, which is used to hold things such as the flow.xml.gz, configuration, state, etc. dataStorage: size: 1Gi # Storage capacity for the FlowFile repository flowfileRepoStorage: size: 3Gi # Storage capacity for the Content repository contentRepoStorage: size: 3Gi # Storage capacity for the Provenance repository. When changing this, one should also change the properties.provenanceStorage value above, also. provenanceRepoStorage: size: 4Gi # Storage capacity for nifi logs logStorage: size: 3Gi   Nifi.properties related to provenance nifi.provenance.repository.directory.default=../provenance_repository nifi.provenance.repository.max.storage.time=10 days nifi.provenance.repository.max.storage.size={{.Values.properties.provenanceStorage}} nifi.provenance.repository.rollover.time=30 secs nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.query.threads=2 nifi.provenance.repository.index.threads=2 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.always.sync=false nifi.provenance.repository.journal.count=16 # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are: # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type nifi.provenance.repository.indexed.attributes= # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository # but should provide better performance nifi.provenance.repository.index.shard.size=400 MB # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved. nifi.provenance.repository.max.attribute.length=65536 ... View more

Folks, We have deployed the nifi with ca via cetic helm chart on a secured k8 cluster with external zookeeper cluster. Now we are facing the issue when nodes are trying to send the hearbeat to cordinator node and results in certificate error something like "nifi path validation failed: java.security.cert.CertPathValidatorException: Path does not chain....." Any help? ... View more