I want to use Nifi Cluster without using ZooKeeper

[#1]

Hello I am trying to setup the Nifi Cluster....is there any way to setup and run clusters without using zookeeper and with which version of nifi [currently using(1.1.2)]?

[#2 · Accepted solution]

@Prakhar Agrawal

With the NiFi 1.x baseline, zookeeper is a fundamental part of the NiFi cluster. It is used to automatically determine the cluster coordinator as well as communication within the cluster.

With the NiFi 0.x baseline, the cluster has an cluster manager which serves a similar role and did not use zookeeper at all. So, if you do not want to use zookeeper, you will have to use a 0.x baseline version of NiFi.

[#3]

Could you please share the steps to configure Nifi 0.x baseline

[#4]

@Thanks Wynner

I have successfully setup cluster using 3 nodes. Now I have one query regarding cluster architecture.

In my flow I use getSFTP processor to get file and for distributing on cluster I am using Remote Process Group. Now I according to my flow only one primary node can get file and keep on their local content repository untill RPG distribute files on different nodes. so is there any way to make this content repository common between all three nodes because right now if in any case my primary node have low memory then what happen with files.

[#5]

@Prakhar Agrawal

I would make a small change to your flow. Use the ListSFTP processor to get a listing of files, then send the list of files to the Remote Process Group (RPG), each node will then receive a listing of different files. Then you can have all three nodes pull files with the GetSFTP processor because they will all be pulling different files.

In answer to your original question though, currently there is no way to share a content repository on one node with all of the other nodes in a cluster,

Also, would you move this to a new question in HCC? Then I'll answer under the new question.