Unable to add a node to Nifi Cluster

[#1]

Hi,

I have a running cluster with 3 nifi nodes, all running an embedded zookeeper. Now I want to add a fourth node which doesn't run an embedded zookeeper. What I did was, on the new node edited the nifi properties, changed nifi.cluster.is.node to true, and added the first 3 nodes zookeeper url to nifi.zookeeper.connect.string. Once done I started nifi, however I get the below error. Posting the full stack trace, this stacktrace keeps on coming infinitely:

1. 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
2. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
3. at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
4. at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
5. at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
6. at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
7. at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
8. at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
9. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
10. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
11. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
12. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
13. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
14. at java.lang.Thread.run(Thread.java:745)

Below is a longer stack-trace right from startup till the error:

1. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures}
2. 2018-11-05 06:09:41,961 INFO [main] org.eclipse.jetty.server.Server jetty-9.4.3.v20170317
3. 2018-11-05 06:09:44,086 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1763ms
4. 2018-11-05 06:09:44,247 INFO [main] org.eclipse.jetty.server.session DefaultSessionIdManager workerName=node0
5. 2018-11-05 06:09:44,248 INFO [main] org.eclipse.jetty.server.session No SessionScavenger set, using defaults
6. 2018-11-05 06:09:44,249 INFO [main] org.eclipse.jetty.server.session Scavenging every 660000ms
7. 2018-11-05 06:09:44,293 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5b49b1df{/nifi-update-attribute-ui-1.7.1,file:///nifi/work/jetty/nifi-update-attribute-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-update-attribute-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-update-attribute-ui-1.7.1.war}
8. 2018-11-05 06:09:44,915 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=511ms
9. 2018-11-05 06:09:44,963 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@11ec2b2f{/nifi-standard-content-viewer-1.7.1,file:///nifi/work/jetty/nifi-standard-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-standard-content-viewer-1.7.1.war}
10. 2018-11-05 06:09:45,009 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=34ms
11. 2018-11-05 06:09:45,041 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@2446be5f{/nifi-image-viewer-1.7.1,file:///nifi/work/jetty/nifi-image-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-media-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-image-viewer-1.7.1.war}
12. 2018-11-05 06:09:46,731 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1430ms
13. 2018-11-05 06:09:46,770 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@71dfca65{/nifi-jolt-transform-json-ui-1.7.1,file:///nifi/work/jetty/nifi-jolt-transform-json-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-jolt-transform-json-ui-1.7.1.war}
14. 2018-11-05 06:09:46,959 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=38ms
15. 2018-11-05 06:09:46,965 INFO [main] org.eclipse.jetty.ContextHandler./nifi No Spring WebApplicationInitializer types detected on classpath
16. 2018-11-05 06:09:47,006 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1d6dc2b8{/nifi,file:///nifi/work/jetty/nifi-web-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-ui-1.7.1.war}
17. 2018-11-05 06:09:47,124 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=57ms
18. 2018-11-05 06:09:47,164 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api No Spring WebApplicationInitializer types detected on classpath
19. 2018-11-05 06:09:47,180 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api Initializing Spring root WebApplicationContext
20. 2018-11-05 06:09:49,601 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties'
21. 2018-11-05 06:09:49,602 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties'
22. 2018-11-05 06:09:49,603 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 148 properties from /nifi/./conf/nifi.properties
23. 2018-11-05 06:09:51,129 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE.Building now...
24. 2018-11-05 06:09:51,437 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 94 properties from system properties and environment variables
25. 2018-11-05 06:09:51,438 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 94 properties.Including precedence overrides effective accessible registry key size is 94
26. 2018-11-05 06:09:51,485 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository using 256 partitions
27. 2018-11-05 06:09:51,507 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 0 records
28. 2018-11-05 06:09:51,527 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0
29. 2018-11-05 06:09:51,533 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 5424567582 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size
30. 2018-11-05 06:09:51,534 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false
31. 2018-11-05 06:09:51,657 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates
32. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures}
33. 2018-11-05 06:09:51,671 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 checkpointed with 0 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1
34. 2018-11-05 06:09:51,755 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected...
35. 2018-11-05 06:09:51,826 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
36. 2018-11-05 06:10:00,725 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected
37. 2018-11-05 06:10:00,727 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
38. 2018-11-05 06:10:01,832 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
39. 2018-11-05 06:10:01,835 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election.
40. 2018-11-05 06:10:01,836 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
41. 2018-11-05 06:10:01,845 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election.
42. 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started
43. 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started
44. 2018-11-05 06:10:04,629 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6149184e{/nifi-api,file:///nifi/work/jetty/nifi-web-api-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.7.1.war}
45. 2018-11-05 06:10:05,255 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=463ms
46. 2018-11-05 06:10:05,287 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath
47. 2018-11-05 06:10:05,290 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@8eb1297{/nifi-content-viewer,file:///nifi/work/jetty/nifi-web-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.7.1.war}
48. 2018-11-05 06:10:05,323 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=20ms
49. 2018-11-05 06:10:05,325 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default
50. 2018-11-05 06:10:05,367 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath
51. 2018-11-05 06:10:05,370 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@646ba7be{/nifi-docs,file:///nifi/work/jetty/nifi-web-docs-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.7.1.war}
52. 2018-11-05 06:10:05,421 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=35ms
53. 2018-11-05 06:10:05,471 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath
54. 2018-11-05 06:10:05,475 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1b6dab22{/,file:///nifi/work/jetty/nifi-web-error-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.7.1.war}
55. 2018-11-05 06:10:05,492 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@2a654167{HTTP/1.1,[http/1.1]}{<node-ip>:8081}
56. 2018-11-05 06:10:05,492 INFO [main] org.eclipse.jetty.server.Server Started @47964ms
57. 2018-11-05 06:10:05,505 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...
58. 2018-11-05 06:10:05,509 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 8082
59. 2018-11-05 06:10:05,587 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
60. 2018-11-05 06:10:05,610 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node:<node-ip>:8081
61. 2018-11-05 06:10:13,959 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
62. 2018-11-05 06:10:13,959 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
63. 2018-11-05 06:10:21,670 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
64. 2018-11-05 06:10:21,670 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
65. 2018-11-05 06:10:21,675 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
66. 2018-11-05 06:10:21,677 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1d2e0826 Connection State changed to SUSPENDED
67. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures}
68. 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
69. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
70. at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
71. at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
72. at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
73. at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
74. at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
75. at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
76. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
77. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
78. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
79. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
80. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
81. at java.lang.Thread.run(Thread.java:745)
82. 2018-11-05 06:10:21,687 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
83. org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
84. at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838)
85. at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
86. at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
87. at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
88. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
89. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
90. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
91. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
92. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
93. at java.lang.Thread.run(Thread.java:745)

[#2]

Siddharth: I am getting same error. I am running nifi in container.Please let me know if you get the answer. Thanks!

[#3]

Hi @Kamal Sehgal , Can you share some useful link about how did u setup the cluster because I am also configuring NiFi cluster with 3 machines (1 Manager and 2 Nodes) but could not be able to configure it yet. My NiFi version is 1.8.0 and using external Zookeeper instead of embedded Zookeeper.

[#4]

Hi @Kamal Sehgal:

What is the issue that you are facing.

Also I found this useful:

https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/

https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/

Its actually easier using an external zookeeper(first link has the steps) . In case you get stuck, let me know.

Also there is no concept of manager node in nifi now. Refer to, https://hortonworks.com/blog/apache-nifi-1-0-0-zero-master-clustering/

[#5]

Hi @siddharth pande, I am also configuring NiFi cluster with 3 machines (1 Manager and 2 Nodes) but could not be able to configure it yet. Can you share some useful link in this regard. My NiFi version is 1.8.0 and using external Zookeeper instead of embedded Zookeeper.