We want to create NiFi Cluster with HA but have only two nodes, and because we have installed 3 Zookeepers on HDP Cluster, we want to use them in NiFi configuration.
Is it possible to configure cluster as I described above? Will we have split brain issues in this configuration?
Yes, I'd avoid 2 nodes for NiFi - nodes tend to vote in a few critical execution paths in a cluster, and a 50/50 one will yield resolution ineffective. Also, consider HDF, it will automate full cluster setup among other things for you.
From my understanding, Zookeeper manages the primary and the coordinator election for NiFi cluster. Hence, the constraint of having an odd number is for Zookeeper nodes and not for NiFi nodes. Am I missing something?
ZK will take care of the Primary and Coordinator role selection. NiFi internally will be voting on e.g. latest flow version to determine which edit is 'the right one'.
FYI: Flow election details:
> NiFi internally will be voting on e.g. latest flow version to determine which edit is 'the right one'.
With clusters, increments of 3,5,7 are magic numbers.
Don't use the HDP Zookeeper, use the HDF Zookeeper.
Follow this install on a separate cluster from HDP
You can install NIFI, Zookeeper, Ranger and Storm from scratch for this cluster.
Timothy, can You please explain why I shouldn't use HDP Zookeeper? What kind of problem I will get if I will use HDP Zookeeper and other modules such as Storm, Kafka and Ranger?
This is the same decision-making which i still don't what to do. I have HDP cluster and 5 NIFI cluster 1.3 (non-HDF) and I wanted to use the HDP zookeeper quorum servers to assignment in NIFI. Why is not recommended to implement it using HDP zooKeeper?