I want to setup the NiFi cluster with at least 3 different machines (1 Manager and 2 nodes). I have followed many approaches which are available over internet but could not be able to configure the cluster successfully. Can any one explain in detail how to set up NiFi cluster with at least 3 different machines (1 Manager and 2 nodes). Any help would be very useful in this regard.
Purpose of this cluster : Huge Data Transfer from Remote source path to HDFS with the highest possible data transfer rate.
Where the install instructions at https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/installing-upgrading-hdf.html not useful? Also, just a callout that NiFi itself is a master-less cluster config. I'm assuming that maybe you want to use the "master" for something like Ambari only?
@Rishabh Singh The NiFi Cluster Manager (NCM) which was a dedicated manger which NiFi nodes connected to no longer exists as of the Apache NIFi 1.x releases. As @Lester Martin stated, all instances of NiFi are capable of acting as the "cluster coordinator" (replacement for the old dedicated NCM). A NiFi cluster requires Zookeeper (ZK). ZK is used to elect one node in your NiFi cluster as the "Cluster Coordinator" and one node as the "primary node". Which node is elected to these roles can change at any time.
@Matt Clarke, I have installed external zookeeper and started it. Then I modified configuration files in Zookeeper and NiFi. After this when I restarted NiFi service then I am able to see the cluster symbol on NiFi canvas but there is only 1 node which is connected (showing 1/1). No other node is getting connected with the cluster. Can you share some useful link or explain how this should be done?
Now that you have one node up and connected to cluster (meaning it is able to send heartbeats to cluster coordinator), you need to look at the nifi-app.log on your other two nodes as they are starting to see why they have not attempted to connect to cluster.
Perhaps they can't talk to zookeeper?
Perhaps they can not resolve the hostname of the elected cluster coordinator for sending heartbeats?