Nifi- Zookeeper - Coordinator Primary Connected

[#1]

Hi All,

I have recently configured 3 Node Nifi cluster Setup, I need some clarification on this.

How zookeeper elected a role for the nodes ?

Whether the role will change for every restar? because

Initially i have seen

Node 1 is Coordinator

Node 2 is Primary

Node 3 is connected state.

But for testing purpose we have restarted the nifi instances but after restart we are seeing the Coordinator ane primary are same nodes.

Sample Image

Please clarify on this,

Is there any option to assign the role? or zookeeper will auto assign the roles?

because of this role change will cause any data loss?

Need your guidance on this.

[#2]

@Pierre Villard Can you help on this too? Why this behaviour? is it safe to run Primary Coordinator on same node?

[#3 · Accepted solution]

@Anishkumar Valsalam

All Nodes will register with ZK to become the cluster coordinator when NiFi cluster is first started. Once all nodes have checked in to ZK or 5 minutes has passed a random node from those who connected will be picked as the cluster coordinator. ZK will also register one node as your primary node.

Once a cluster coordinator has been elected, all nodes will start sending heartbeats directly to that node. The cluster coordinator assumes the role of disconnecting nodes from the cluster who do not send heartbeats and reconnecting nodes who heartbeat later after previously being disconnected.

Nodes in a cluster also heartbeat with ZK. If either the primary node or cluster coordinator fails to heartbeat, another connected node(s) at random is elected to assume those roles.

There is no ability for users to manually assign either of these roles to a specific node in a cluster.

Thanks,

Matt

[#4]

Role change does not cause data loss.

Every node in a cluster runs the same dataflow and works on its own set of FlowFiles. Processor components added to the canvas and configured to run on primary node only will run on the currently elected primary node. So when primary node assignment changes the primary node only configured processors are stopped on the old primary node and started on the new. So be mindful of what processors are set to run on primary node only. While this will not result in data loss, it could result in data being stalled in a dataflow.

[#5]

@Matt Clarke Thanks much. so there is no harm when one nodes have 2 roles.

Example :

Node 1 is Coordinator, Primary.

Node 2 is connected

Node 3 is connected

[#6]

@Anishkumar Valsalam

No harm at all having one node service both roles. It is very common to see that.

Matt

[#7]

Thanks Much.