NiFi Large FlowFile Attributes

[#1]

I understand NiFi stores its FlowFile attributes in memory and large attributes are not recommended because it takes up space in the JVM. What I am having trouble grasping is the consequences. Can someone explain different use cases what could happen and also, how long those attributes are stored in memory for?

[#2 · Accepted solution]

Every flowfile is held in memory for as long as the flowfile is in the flow, until it reaches a DROP event[1]. This means, that if you're writing a large attribute (such as a json) this will eventually add up to a large heap usage.

Keep in mind that the attributes also get stored in data provenance so there's also an impact there.

Biggest consequence to this a risk of NiFi going OOM, then failing to load the flowfile repository on startup.

If the flowfile repo is too large it can also take longer for NiFi to start up, or also go OOM on startup while trying to load the repositories.

This is also touched on the Anti-patterns video[2]

[1]https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#provenance_events
[2]https://www.youtube.com/watch?v=RjWstt7nRVY