Skip to main content
manishg
manishg
Gold ⭐️
January 29, 2024
Solved

disk io operations going up with volatile repositories

I have set following:

  1. nifi.flowfile.repository.implementation: org.apache.nifi.controller.repository.VolatileFlowFileRepository
  2. nifi.provenance.repository.implementation: org.apache.nifi.provenance.VolatileProvenanceRepository

With this, I expect disk io operations to go down(checked with nmon). But its just opposite and disk writes have gone up multiple times.

Why so?

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
January 29, 2024

@manishg 

Same about of flowFiles per second processing after switching to the Volatile repositories?
Perhaps having FlowFile and provenance repositories in memory allows for faster processing of FlowFIles resulting in more read and writes to the content_repository which contains the actual content of each FlowFile.

If your NiFi should crash or restart you will lose everything in your volatile repositories.  The FlowFile repository holds all the FlowFile metadata for the FlowFiles currently being processed through your dataflows.   This means data loss in such events.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

View in thread

1 reply

MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
January 29, 2024

@manishg 

Same about of flowFiles per second processing after switching to the Volatile repositories?
Perhaps having FlowFile and provenance repositories in memory allows for faster processing of FlowFIles resulting in more read and writes to the content_repository which contains the actual content of each FlowFile.

If your NiFi should crash or restart you will lose everything in your volatile repositories.  The FlowFile repository holds all the FlowFile metadata for the FlowFiles currently being processed through your dataflows.   This means data loss in such events.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate
Terms & ConditionsAccessibility statement