question Re: Can NIFI storage be extended? in Support Questions

question Re: Can NIFI storage be extended? in Support Questions https://community.cloudera.com/t5/Support-Questions/Can-NIFI-storage-be-extended/m-p/306356#M222807 <a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/74983">@dzbeda</a>  Can you share a little more about your use case? NiFi does not expire data that is actively queued within connections between components added to the NiFi canvas.   So I am a bit curious on the "I don't want to lose data" statement you made. It is true that during times of "connectivity issues between the sites" that  NiFi FlowFile may accumulate within the connection queues resulting in more storage being needed to hold that queued data while you wait for the connectivity to restore, but still not a concern for "data loss" unless your ingest is using some unconfirmed transfer protocol like UDP.  NiFi's Site-To-Site protocol used by the Remote Process Groups uses a two phase commit to avoid dataloss. Backpressure settings on each connection can control how many FlowFiles can queue before the component feeding FlowFiles into the connection is o longer allowed to execute. So in an extended outage or high volume, backpressure could end up being applied to all connection from last component in your dataflow to the first component in your dataflow.   Default object thresholds are (10,000 FlowFiles or 1 GB of content size). Keep in mind these are soft limits.  Not advisable to simply set backpressure to some much larger value. I recommend reading following article: <A href="https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166" target="_blank">https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166</A> As far as what happens when the content repo(s) (NiFi allows you to configure multiple content repos per NiFi instance) are full, NiFi simply can not generate any new content.  So any component that tries to create new content (can be at ingest or via some processor that modifies the content of an existing FlowFile) will simply fail went it tries to do so with an out of disk space exception. This does not mean dataloss (unless as I mentioned your ingest or egress uses an unconfirmed protocol).  The component will simply try again until it is successful once disk space becomes available (For example when connectivity returns and data can be pushed out). Using good protocols would result on data remaining on source once backpressure is applied all the way back to your ingest based components. NiFi archiving has nothing to do with how long FlowFiles are kept in NiFi's dataflow connections.  Archiving holds FlowFiles after they have successfully been removed (reached point of auto-termination in a dataflow.   Archiving allows you to view old FlowFiles no longer queued or replay a FlowFiles from any point in your dataflow.  However, there is no bulk replay capability, so not useful for that. <A href="https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418" target="_blank">https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418</A> Hope this helps, Matt Mon, 23 Nov 2020 13:34:11 GMT MattWho 2020-11-23T13:34:11Z