Hi Folks,
I've been facing an issue with nifi pod restarting only when persistence is enabled, Below is the app.log excerpt.
I doubt this could be related to provenance. Can someone help?
APP.log
2023-01-20 12:47:20,598 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loading Application Properties [/opt/nifi/nifi-current/./conf/nifi.properties]
2023-01-20 12:47:29,181 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates
2023-01-20 12:47:29,181 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 2 milliseconds
2023-01-20 12:47:29,192 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 checkpointed with 0 Records and 0 Swap Files in 11 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1
2023-01-20 12:47:29,991 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 117 properties from system properties and environment variables
2023-01-20 12:47:29,991 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 117 properties. Including precedence overrides effective accessible registry key size is 117
2023-01-20 12:47:31,323 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository
2023-01-20 12:47:31,328 INFO [main] o.apache.nifi.controller.FlowController Creating Provenance Repository [org.apache.nifi.provenance.WriteAheadProvenanceRepository]
2023-01-20 12:47:31,598 INFO [main] o.a.n.p.store.WriteAheadStorePartition After recovering ../provenance_repository, next Event ID to be generated will be 0
2023-01-20 12:47:31,754 INFO [main] o.a.n.p.index.lucene.LuceneEventIndex Will avoid re-indexing Provenance Events because the newest index is defunct, so it will be re-indexed in the background
2023-01-20 12:47:31,806 INFO [main] o.apache.nifi.controller.FlowController Creating Content Repository [org.apache.nifi.controller.repository.FileSystemRepository]
2023-01-20 12:47:31,827 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 9126805504 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size
2023-01-20 12:47:31,842 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false
2023-01-20 12:49:29,376 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 checkpointed with 0 Records and 0 Swap Files in 182 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1
2023-01-20 12:49:55,078 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2023-01-20 12:49:55,188 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loading Application Properties [/opt/nifi/nifi-current/./conf/nifi.properties]
2023-01-20 12:49:55,198 INFO [main] org.apache.nifi.NiFi Application Properties loaded [142]
2023-01-20 12:49:55,291 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 40217
2023-01-20 12:49:55,314 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2023-01-20 12:49:55,390 INFO [main] org.apache.nifi.nar.NarUnpacker Expanding 122 NAR files with all processors...
Persistence settings
configStorage:
size: 100Mi
authconfStorage:
size: 100Mi
# Storage capacity for the 'data' directory, which is used to hold things such as the flow.xml.gz, configuration, state, etc.
dataStorage:
size: 1Gi
# Storage capacity for the FlowFile repository
flowfileRepoStorage:
size: 3Gi
# Storage capacity for the Content repository
contentRepoStorage:
size: 3Gi
# Storage capacity for the Provenance repository. When changing this, one should also change the properties.provenanceStorage value above, also.
provenanceRepoStorage:
size: 4Gi
# Storage capacity for nifi logs
logStorage:
size: 3Gi
Nifi.properties related to provenance
nifi.provenance.repository.directory.default=../provenance_repository
nifi.provenance.repository.max.storage.time=10 days
nifi.provenance.repository.max.storage.size={{.Values.properties.provenanceStorage}}
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=400 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
