Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi Storage

Highlighted

Nifi Storage

New Contributor

Hello,


In Nifi properties file, there is a field nifi.provenance.repository.max.storage.time=24 hours.


I want to know what maximum time I can put for this field. Default value is 24 hours.


Thanks

Vivek

1 REPLY 1

Re: Nifi Storage

Master Guru

@Vivek Thakur

You can put whatever value you like in there.
-
for example:

48 hours
30 days

90 days

etc....
-
Setting a value of say 90 days here does not guarantee that you will even have 1 day of provenance history. The amount of provenance history retained is going to be capped by another Provenance configuration and rate at which your dataflow(s) are producing provenance events.
-

nifi.provenance.repository.max.storage.size

The maximum amount of data provenance information to store at a time. The default value is 1 GB. The Data Provenance capability can consume a great deal of storage space because so much data is kept. For production environments, values of 1-2 TB or more is not uncommon.

-

The more processors in use and the higher the number of FlowFile being processed through these processors will equate to higher number of produced provenance events.

-

If you are looking to store large amounts of provenance events, you may want to consider adjusting the defaults on the below setting as well to have better provenance query performance:

-

nifi.provenance.repository.index.shard.size

The repository uses Apache Lucene to performing indexing and searching capabilities. This value indicates how large a Lucene Index should become before the Repository starts writing to a new Index. Large values for the shard size will result in more Java heap usage when searching the Provenance Repository but should provide better performance. The default value is 500 MB. However, this is due to the fact that defaults are tuned for very small environments where most users begin to use NiFi. For production environments, it is advisable to change this value to 4 to 8 GB. Once all Provenance Events in the index have been aged off from the "event files," the index will be destroyed as well. Note: this value should be smaller than (no more than half of) the nifi.provenance.repository.max.storage.size property.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link