https://community.cloudera.com/t5/Support-Questions/NiFi-Large-FlowFile-Attributes/m-p/357393#M237574 <P>Every flowfile is held in memory for as long as the flowfile is in the flow, until it reaches a DROP event[1]. This means, that if you're writing a large attribute (such as a json) this will eventually add up to a large heap usage.</P><P>Keep in mind that the attributes also get stored in data provenance so there's also an impact there.</P><P>&nbsp;</P><P>Biggest consequence to this a risk of NiFi going OOM, then failing to load the flowfile repository on startup.</P><P>If the flowfile repo is too large it can also take longer for NiFi to start up, or also go OOM on startup while trying to load the repositories.</P><P>&nbsp;</P><P>This is also touched on the Anti-patterns video[2]</P><P><BR />[1]<A href="https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#provenance_events" target="_blank">https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#provenance_events</A><BR />[2]<A href="https://www.youtube.com/watch?v=RjWstt7nRVY" target="_blank">https://www.youtube.com/watch?v=RjWstt7nRVY</A></P> Thu, 10 Nov 2022 18:58:51 GMT bbahamondes 2022-11-10T18:58:51Z https://community.cloudera.com/t5/Support-Questions/NiFi-Large-FlowFile-Attributes/m-p/357388#M237572 <P>I understand NiFi stores its FlowFile attributes in memory and large attributes are not recommended because it takes up space in the JVM. What I am having trouble grasping is the consequences. Can someone explain different use cases what could happen and also, how long those attributes are stored in memory for?</P> Thu, 10 Nov 2022 17:20:45 GMT https://community.cloudera.com/t5/Support-Questions/NiFi-Large-FlowFile-Attributes/m-p/357388#M237572 drewski7 2022-11-10T17:20:45Z https://community.cloudera.com/t5/Support-Questions/NiFi-Large-FlowFile-Attributes/m-p/357393#M237574 <P>Every flowfile is held in memory for as long as the flowfile is in the flow, until it reaches a DROP event[1]. This means, that if you're writing a large attribute (such as a json) this will eventually add up to a large heap usage.</P><P>Keep in mind that the attributes also get stored in data provenance so there's also an impact there.</P><P>&nbsp;</P><P>Biggest consequence to this a risk of NiFi going OOM, then failing to load the flowfile repository on startup.</P><P>If the flowfile repo is too large it can also take longer for NiFi to start up, or also go OOM on startup while trying to load the repositories.</P><P>&nbsp;</P><P>This is also touched on the Anti-patterns video[2]</P><P><BR />[1]<A href="https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#provenance_events" target="_blank">https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#provenance_events</A><BR />[2]<A href="https://www.youtube.com/watch?v=RjWstt7nRVY" target="_blank">https://www.youtube.com/watch?v=RjWstt7nRVY</A></P> Thu, 10 Nov 2022 18:58:51 GMT https://community.cloudera.com/t5/Support-Questions/NiFi-Large-FlowFile-Attributes/m-p/357393#M237574 bbahamondes 2022-11-10T18:58:51Z