NiFi Repository - Typical Disk Usage Ratios among the repositories

[#1]

Do we have any experience on typical disk usage rations for each of the repositories (Flow file, content, and provenance)? E.g. if Content requires 200 GB of storage, the provenance and flow file would require 20 GB (for typical flows)?

Trying to use this information to decide how best to slice of a NiFi server which has 12 local drives. E.g. 8 drives allocated for Content, 2 for flow file, and 2 for provenance.

Appreciate any thoughts!

[#2 · Accepted solution]

There is no direct correlation between the size of the content repository and the provenance repository. The size the content repository will grow to is directly tied to the amount of unique content that is currently queued on the NiFi canvas. If archive is enabled the amount of content repository space consumed will depend on the archive configuration settings in the nifi.properties file.

nifi.content.repository.archive.max.retention.period=12 hours

nifi.content.repository.archive.max.usage.percentage=75%

nifi.content.repository.archive.enabled=true

As you can see from the above archive will try to retain 12 hours of archived content (archived content being content that is no longer associated to an existing queued FlowFile on within any dataflow on the graph. This does not guarantee that there will be any archive or that the content repository will not grow beyond 75% disk utilization. Content still actively associated to queued FlowFiles will remain in the Content repository. So it is important to build in back pressure in to dataflows where there is concern that large backlogs could trigger disk to fill to 100%. Should Content repo fill to 100% corruption will not occur. New FlowFiles will not be able to be created until free space is available. This is likely to produce a lot of errors in the flow (anywhere content is modified/written).

Provenance repository size is directly related to the number of FlowFiles and the number of event generating processors those events pass through on the NiFi canvas. In the case of disk utilization here, it is very controlled by setting in the nifi.properties file:

nifi.provenance.repository.max.storage.time=7 days

nifi.provenance.repository.max.storage.size=50 GB

With the above settings, NiFi will try to retain 7 days of provenance events on every FlowFile that it processes, but will start rolling off the oldest events once the max storage exceeds 50 GB.

It is important to understand that the 75% and 50GB are soft limits and should never be set to 100% or the exact size of the disk.

FlowFile Repository and database repository each remain relatively small. The FlowFile repository is the most important repo if all. It should be isolated on a separate disk/partition that is not shared with any other process that may fill it. allowing the FlowFile repository disk to fill to 100% can lead to database corruption and lost data. for a 200 GB Content repository, a ~25 GB FlowFile repo should be enough. The database repository contains the user and change history DBs. The user db will remain 0 bytes in size for NiFi instances running http (non-secure). For those instances running https (Secure), the user db will track all users who log in to the UI. The change history db is tied to the little clock icon in the upper right corner NiFi tool bar. It keeps track of all changes made on the NiFi graph/canvas. It also stays relatively small. A few GB of space should be plenty to store a considerable number of changes.

[#3]

for you scenario with 12 disks (assuming all disk are 200 GB)

You can specify/define multiple Content repos and multiple Provenance repos; however, you can only define one FlowFile repository and one database repository.

- 8 disks for Content repos:

- /cont_repo1 <-- 200 GB

- /cont_repo2 <-- 200 GB

- /cont_repo3 <-- 200 GB

- /cont_repo4 <-- 200 GB

- /cont_repo5 <-- 200 GB

- /cont_repo6 <-- 200 GB

- /cont_repo7 <-- 200 GB

- /cont_repo8 <-- 200 GB

- 2 disks for Provenance repos:

- /prov_repo1 <-- 200 GB

- /prov_repo2 <-- 200 GB

- 1 disk split into multiple partitions for:

- /var/log/nifi-logs/ <-- 100 GB

- OS partitions <-- split amongst other Standard OS (/tmp, /, etc...)

- 1 disk split into multiple partitions for:

- /opt/nifi <-- 50 GB

- /flowfile_repo/ <-- 50 GB

- /database_repo/ <-- 25 GB

- /opt/configuration-resources <-- 25 GB (this will hold any certs, config files, extras your NiFi processors/ dataflows may need).

[#4]

Is there any reason for which the provenance repository would not start deleting files once it has reached the maxStorage limit ?

In my case even though nifi.provenance.repository.max.storage.size=10 GB, its not deleting anything. But when checking the files in provenance, it seems its following the days limit as the oldest file is 7 days old. There are a lot of toc tmp files under the folder.

nifi.provenance.repository.max.storage.time=7 days

Anything I should look into for this ?

[#5]

@Scorpy257 As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.