Nifi content repository not cleaning up and keeps accumalating over time

[#1]

Nifi content repository works in sync with the Nifi Status bar for sometime after a restart. After that, it starts growing gradually and comes down again only on a restart.

I have a mix of small and large files and are merging files to create big files in the flow. I am potentially hitting a bug https://issues.apache.org/jira/browse/NIFI-6150 which is not working for probably the below mentioned three scenarios:

1) Having both small and big files.

2) Merging small files to create big files. I have a few big files which only go after a restart.

3) Merge content without any failures holds onto flow files for days after the flow file has left flow.

Data flow for this Nifi is 24 TB per day and restart is causing huge lags.

[#2]

Dear,

I experienced the same situation but after a migrtion from 1.11.4 to 1.13.2. Have you solve your problem?

Jean-Louis

[#3]

@Vinayakmkmishra 

NiFi FlowFile content claims can contain the content for 1 to many FlowFile.  A content claim can not be deleted from the content repository until all FlowFiles reference that content claim are no longer queued anywhere in the dataflow.  So it is possible a 1 byte content FlowFile somewhere in your dataflow(s) could be holding up a claim of a much larger size.

You can never expect yoru content repository usage to match up with the cumulative queued content size reported in the NiFi UI.  What is summed up for you in the UI is representative of the FlowFiles still queued through out your NiFi dataflow(s) and not representative of the size of the many content claims so various bits of content may exist as part of.

You may find the following article about the content repository helpful as well:
https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Arc...

Above being said, there are some known bugs that can prevent the content repository cleanup from working, but you have not shared what NiFi version you are using.

1. NIFI-6150
2. NIFI-6236
3. NIFI-6846
4. NIFI-7469
5. NIFI-7992

I recommend upgrading to latest NiFi release which resolves all the above issues.

If you found this addressed your query, please take a moment to login and click accept on all solutions that helped you.
Thank you,

Matt

[#4]

I think I figured out with this issue on my side and I share my results with you.

I deal with very small and very huge contents, the default values for properties hereafter are not appropriate (Claim size retained ...), so I decrease drastically the values as follows:

• nifi.content.claim.max.appendable.size=10 MB   ==> 5MB
• nifi.content.claim.max.flow.files=100 ==> 50
• nifi.content.repository.archive.max.retention.period=12 hours ==> 2 hours
  

Until now, the platform doesn't freeze anymore.

Jean-Louis