@Onkar_Gagre  1. What is the CPU and Memory usage of your NiFi instances when the QueryRecord processor is stopped? 2. How is your QueryRecord processor configured to include scheduling and concurrent task configurations? What other processors were introduced as part of this new dataflow? 3. What does disk I/O look like while this processor is running?   NiFi documentation does not mention any CPU or Memory specific resource considerations when using this processor. Thanks, Matt ... View more

@ajignacio That was a big jump from version 1.9.x to 1.16.x of NiFi. NiFi's data provenance stores, for a configurable amount of time, information about NiFi FlowFiles as they traverse the various processors in your dataflow(s).  Over the releases of NiFi both improvements and new implementations of provenance have been introduced.  The original version of provenance was org.apache.nifi.provenance.PersistentProvenanceRepository which has since been deprecated in favor of a better performing provider class org.apache.nifi.provenance.WriteAheadProvenanceRepository which is the new default.   The following properties from the nifi.properties file are used to configure the provenance repository: nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository nifi.provenance.repository.directory.default=./provenance_repository nifi.provenance.repository.max.storage.time=30 days nifi.provenance.repository.max.storage.size=10 GB. (use to be 1 GB) nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.query.threads=2 nifi.provenance.repository.index.threads=2 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.always.sync=false nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID nifi.provenance.repository.indexed.attributes= nifi.provenance.repository.index.shard.size=100 MB nifi.provenance.repository.max.attribute.length=65536 nifi.provenance.repository.concurrent.merge.threads=2 nifi.provenance.repository.warm.cache.frequency= For details on these properties, here is Apache NiFi documentation section: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#provenance-repository   The good news is that data provenance retention has no direct relationship to the active FlowFiles traversing your dataflow(s) currently.  This means that you can shutdown your NiFi, purge the contents of the current <path to>/provenance_repository directory, adjust the configuration properties as you want, and then restart your NiFi.   NiFi will build a new provenance repository on startup.  Considering that NiFi only provides limited configurable space (1GB original default to 10GB current default) and age (30 days) as the defaults, you would not be losing much if you were to reset. I am also concerned that the path in the error suggests you also created your original provenance_repository within a subdirectory of the FlowFile_repository which I would not recommend.  I would strongly suggest not writing the contents of any one of the four NiFi repositories within each other. Considering the flowfile_repository and content_repository are the two most important repositories for tracking your actively being processed FlowFiles in your dataflow(s), I suggest these each be on their own path and reside on dedicated disk backed by RAID to avoid data loss in the event of a disk failure.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Sinchan  You'll want to inspect the configuration of the the following properties in the nifi.properties configuration file:   When you configure a secure NiFi configuration, these properties must be configured.  If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Mohamed_Shaaban  I recommend starting a new community question with the details specific to your setup.  This allows the community to address/assist with your specific setup versus comparing your issue to what was shared in this post. Thanks, Matt ... View more