question Why only flowfile repository disk is getting full and other repos are not? in Support Questions

Why only flowfile repository disk is getting full and other repos are not?

hegdemahendra — Sat, 11 May 2024 09:13:56 GMT

Hello experts,

I am facing an issue in one of the Nifi server where we have multiple consume eventhub flows.

The flow file repository disc is getting full but content and provenance repos are not.

Have attached the screen shot of all repos usage and content of flowfile repo.

journals folder is occupying very large amount of data.

nifi.properties (related to flofile repo):
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.directory=/flowfile
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.retain.orphaned.flowfiles=true
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog

Can anyone help me understand what is the issue? how to resolve this?

Thanks,

Mahendra

Re: Why only flowfile repository disk is getting full and other repos are not?

hegdemahendra — Tue, 14 May 2024 08:13:15 GMT

@MattWho - would appreciate if you have any comment on this issue. Thanks in advance.

Re: Why only flowfile repository disk is getting full and other repos are not?

MattWho — Mon, 20 May 2024 21:01:09 GMT

Hello @hegdemahendra

Always very helpful if you include the exact version of Apache NiFI, Cloudera HDF, or Cloudera CFM being used.

My guess here would be one or both of the following:

You have multiple FlowFiles all pointing at the same content claims queued in connections within your dataflow(s) on the canvas. As long as a FlowFile exists on the canvas it will exist in flowfile_repository. Users should avoid leaving FlowFiles queued in connection on NiFi. Some users tend to allow FlowFile to accumulate at stopped processor components rather then auto-terminate them. Even if a FlowFile does not have any content its FlowFile attributes/metadata still consume disk space.
You are extracting content from your FlowFiles into FlowFile attributes resulting in large FlowFile attribute/metadata being stored in the flowfile_repository. Dataflow designers should avoid extracting large amounts flowfile content in to the FlowFile's attributes. Instead try to build dataflows and utilize components that read content from the FlowFile's content instead of from FlowFile attributes.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt