Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Clustering NiFi with two nodes and using HDP Zookeeper.

Highlighted

Clustering NiFi with two nodes and using HDP Zookeeper.

Contributor

Hello,

We want to create NiFi Cluster with HA but have only two nodes, and because we have installed 3 Zookeepers on HDP Cluster, we want to use them in NiFi configuration.

Is it possible to configure cluster as I described above? Will we have split brain issues in this configuration?

Regards,

Ramil.

7 REPLIES 7

Re: Clustering NiFi with two nodes and using HDP Zookeeper.

Yes, I'd avoid 2 nodes for NiFi - nodes tend to vote in a few critical execution paths in a cluster, and a 50/50 one will yield resolution ineffective. Also, consider HDF, it will automate full cluster setup among other things for you.

Re: Clustering NiFi with two nodes and using HDP Zookeeper.

Hi @Andrew Grande

From my understanding, Zookeeper manages the primary and the coordinator election for NiFi cluster. Hence, the constraint of having an odd number is for Zookeeper nodes and not for NiFi nodes. Am I missing something?

Thanks

Re: Clustering NiFi with two nodes and using HDP Zookeeper.

ZK will take care of the Primary and Coordinator role selection. NiFi internally will be voting on e.g. latest flow version to determine which edit is 'the right one'.

Re: Clustering NiFi with two nodes and using HDP Zookeeper.

Cloudera Employee

FYI: Flow election details:

> NiFi internally will be voting on e.g. latest flow version to determine which edit is 'the right one'.

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#flow-election

Re: Clustering NiFi with two nodes and using HDP Zookeeper.

Super Guru

With clusters, increments of 3,5,7 are magic numbers.

Don't use the HDP Zookeeper, use the HDF Zookeeper.

Follow this install on a separate cluster from HDP

http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/system-requir...

You can install NIFI, Zookeeper, Ranger and Storm from scratch for this cluster.

Re: Clustering NiFi with two nodes and using HDP Zookeeper.

Contributor

Timothy, can You please explain why I shouldn't use HDP Zookeeper? What kind of problem I will get if I will use HDP Zookeeper and other modules such as Storm, Kafka and Ranger?

Re: Clustering NiFi with two nodes and using HDP Zookeeper.

New Contributor

This is the same decision-making which i still don't what to do. I have HDP cluster and 5 NIFI cluster 1.3 (non-HDF) and I wanted to use the HDP zookeeper quorum servers to assignment in NIFI. Why is not recommended to implement it using HDP zooKeeper?

Don't have an account?
Coming from Hortonworks? Activate your account here