@ChampagneM12
When you install a NiFi cluster, you start with a blank canvas. So there is no data ingestion at first. The user must construct data flow(s) to meet their individual use cases as I am sure you know. Handling data ingestion through an outage is handled through your implementation.
Lets assume you are ingesting from Kafka in to NiFi since you mentioned you use Kafka. You would likely start that dataflow with a ConsumeKafka processor. Le's also assume you have a 3 node NiFi cluster and the Kafka topic you are consuming from has 12 partitions.
Since all nodes in your cluster will be executing the consumeKafka processor, each will be a consumer of that topic. With a single concurrent task (default) configured on the ConsumeKafka, each of those 3 NiFi node's consumeKafka will be assigned 4 partitions each. If you were to set the Concurrent tasks to 4, then you now have a total of 12 consumers (one for each Kafka partition). Now lets assume one of your NiFi nodes goes down, Kafka will see a drop in the number of consumers from 12 to 8 and rebalance. So consumption will continue with some of those consumers now being assigned multiple partitions until the down NiFi node comes back on line.
That is just one scenario.
In the case of using a NiFI Listen type processor (example: ListenTCP). This starts a TCP socket listener on each node in the NiFi cluster on the same port. In this case it would be the client or some external mechanism that would need to handle failover to a different node in the event a NiFi node goes down. This is typically handled with an external load balancer which distributes data to all the NiFi nodes or switches to a different node when a node goes down.
In the use case of something like ListSFTP, this processor would be configured to run on "primary node" only. Zookeeper is responsible for electing a primary node and a cluster coordinator in a NiFi cluster. NiFi processor components like ListSFTP are designed for primary node execution only and store state on the data listed in cluster state (within zookeeper). If the current elected primary node goes down, another node in the NiFi cluster is elected the new primary node and the primary node only configured processors are started non that new node. Last recorded state for that component reported to ZK by the previous primary node is pulled from ZK to the new primary node processor and it picks up listing from there. Again you have redundancy.
The only place in NiFi were you can have data delay, is when a NiFi node goes down while it still has active data in its connection queue(s). Other nodes will not have access to that data on the other down node to take over work on it. It will remain in that node's flowfile and content repositories until that node has been restored and can continue processing on that queued FlowFiles. So it is important to protect those two NiFi repositories using RAID configured drives. You can minimize impact in such cases through good flow design and use of back pressure to limit amount of FlowFiles that can queue on a NiFi node.
Also keep in mind that while the flowfie and content repositories are tightly coupled to the flow.xml.gz, these items are not tightly coupled to a specific piece of hardware. You can stand up an entirely new node for you cluster and move the flow.xml.gz, content repo and flowfile repo on to that node before starting it and that new node will continue processing the queued FlowFiles.
Hope this helps,
Matt
