@Vivek Thakur
You can put whatever value you like in there.
-
for example:
48 hours
30 days
90 days
etc....
-
Setting a value of say 90 days here does not guarantee that you will even have 1 day of provenance history. The amount of provenance history retained is going to be capped by another Provenance configuration and rate at which your dataflow(s) are producing provenance events.
-
nifi.provenance.repository.max.storage.size
The maximum amount of data provenance information to store at a time. The default value is 1 GB. The Data Provenance capability can consume a great deal of storage space because so much data is kept. For production environments, values of 1-2 TB or more is not uncommon.
-
The more processors in use and the higher the number of FlowFile being processed through these processors will equate to higher number of produced provenance events.
-
If you are looking to store large amounts of provenance events, you may want to consider adjusting the defaults on the below setting as well to have better provenance query performance:
-
nifi.provenance.repository.index.shard.size
The repository uses Apache Lucene to performing indexing and searching capabilities. This value indicates how large a Lucene Index should become before the Repository starts writing to a new Index. Large values for the shard size will result in more Java heap usage when searching the Provenance Repository but should provide better performance. The default value is 500 MB. However, this is due to the fact that defaults are tuned for very small environments where most users begin to use NiFi. For production environments, it is advisable to change this value to 4 to 8 GB. Once all Provenance Events in the index have been aged off from the "event files," the index will be destroyed as well. Note: this value should be smaller than (no more than half of) the nifi.provenance.repository.max.storage.size property.
-
Thank you,
Matt
-
If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link
