Is Nifi fault-tolerant against machine failures?

[#1]

Hi,

I have a question regarding Nifi:

1. In cluster mode, if a node goes down due to disk-failures etc, will there be data loss on Apache Nifi cluster provided you cannot recover the node?

2. Is there a way to store all repositories/data on a remote location (like we store state on Zookeeper) so if a host goes down, we dont loose any data? Is there a possibility of replication like in Kafka if you loose a Broker, you dont loose any data since it is replicate on other Brokers?

Thanks

Obaid

[#2]

@Obaid Salikeen

Check this article as well as its part 2.

https://community.hortonworks.com/articles/8607/how-to-create-nifi-fault-tolerance-using-multiple.ht...

and

https://community.hortonworks.com/articles/8631/how-to-create-nifi-fault-tolerance-using-multiple-1....

[#3]

@Obaid Salikeen

To answer your question number 1, the answer is you wont have data loss because for nifi you do use RAID 10 so one disk failure should not cause a data loss.

To answer your question your number 2, NiFi does not replicate data like Kafka. If you lose a node, then flow can be directed to a available node however, flowfile queued for the failed node will either wait until the node comes up or the flowfile is manually is sent to another working node.

Here is how to setup clustering

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering

Also, if you read the links I shared above, they will clear any confusion you may have.

[#4 · Accepted solution]

Thanks mqureshi,

As I understand from your comments, Nifi dosent have a built-in fault-tolerant mechanism. Meaning if one node goes down then there has to be a manual intervention to prevent data loss/redirect flow etc? To put it another way, if you dont have a special RAID 10 setup then you will loose data if a node goes down?

Thanks for the links, I will certainly go through them.

Thanks

[#5]

@Obaid Salikeen

You can have a nifi cluster and you will not lose data. When a node goes down, NiFi cluster manager will route the data to another node. However, queued data for the failed node will still be queued for failed node. Only that data must be manually sent over to the live node in the cluster or just bring the failed node up. Any new data, will automatically be routed to other nodes in the cluster by NiFi Cluster Manager (NCM). There is no manual intervention required. Queued data is usually very small, around 1 second of data or even less.

Not only can you run NiFi in a cluster to have redundancy but you can even have different NiFi clusters across different data centers, so in case of a data center failure, you are still able to ingest data.

That's why the two links I shared earlier are very important as they describe the architecture you will need to implement NiFi cluster in a simple data center (one cluster scenario) as well as multiple clusters in multiple data centers.

[#6]

@Obaid Salikeen - Please accept the answer if it helped you.