Adding a node to a NIFI cluster with embedded Zookeeper

[#1]

Hi,

We have a 3 node nifi cluster running, using embedded Zookeeper. Now I need to add a new node to the cluster. What is the procedure for the same? Will I have to bring down all the nodes, since I need to add an entry to the /nifi/conf/zookeeper.conf(server=) and also to the list of zookeeper servers in /nifi/conf/nifi.properties(nifi.zookeeper.connect.string). As far as I understand, these both will have to be edited on the existing 3 as well as the new box and then the nifi server restarted. But dont think this is the right way, since each time the cluster will have to be brought down. Can you please advice?

[#2]

@siddharth, i don't think you need to bring down the cluster. I had tried below scenario and works properly.

Had 2 node cluster. we have to do all cluster related configuration in nifi.properties and provide zookeeper connect string (as provided in your earlier node for property nifi.zookeeper.connect.string) so that this node will get to know who is cluster coordinator or primary node.

[#3]

@ashok.kumar

Just to confirm: You edited the nifi.zookeeper.connect.string in the existing 2 node's nifi.properties file and also added the same in the new node.

Then you would have edited the zookeeper.conf on the 2 existing boxes and also on the new box

Lastly you would have started nifi on the new node.

This made it work?

[#4]

@siddharth pande

I think above reply answers your question. Still if you have any doubt or confusion, let us know.

[#5]

@ashok.kumar: Thanks Ashok, my query is answered. On another note ideally in how many nodes should I keep the embedded zookeeper server running. If I have a six node cluster, in how many should I keep the embedded zookeeper on.

[#6]

@siddharth pande ideally we use three node to be considered as part of zookeeper server.

[#7]

@ashok.kumar: I folllowed the above steps, i.e just adding the zookeper string and enabling is cluser in nifi properties. When I start the server I get the below error at a very fast rate:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre}

2018-11-02 09:22:09,009 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up

org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss

at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838)

at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)

at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)

at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Any pointers?

[#8]

@siddharth , no, i did not edit nifi.zookeeper.connect.string in existing 2 nodes' nifi.properties file. I just provided zookeeper connection string to new node and also configured cluster properties like enabling cluster.

With this, once you start node it check if it has to start in cluster mode and then looks for any leader is there. In this case a leader will be available and to that leader it will send its heartbeat.

[#9]

@ashok.kumar: Got it, thanks, just one last thing. All online tutorial require us to edit the zookeeper.conf file with some thing like:

<node-ip>:2888:3888, what changes did you make with regard to this file?

[#10]

@siddharth pande you need to edit zookeeper.conf only if you want this node to be part of embedded zookeeper server. Since i just need my node to be part of cluster and i did not want my node to part of zookeeper server and hence i did not modify zookeeper.properties and also had set nifi.state.management.embedded.zookeeper.start as false.

If i want to add my current node as part of zookeeper server then i have to enable embedded zookeeper in nifi.properties and also modify zookeeper.properties to add my current node as another zookeeper server, And also i can add this node as part of zookeeper connection string.

If we don't want our current node to be part of zookeeper server then current node will be only zookeeper client and for that only connection string is enough to connect to existing zookeeper server.