Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to add a node to Nifi Cluster

Unable to add a node to Nifi Cluster

Explorer

Hi,

I have a running cluster with 3 nifi nodes, all running an embedded zookeeper. Now I want to add a fourth node which doesn't run an embedded zookeeper. What I did was, on the new node edited the nifi properties, changed nifi.cluster.is.node to true, and added the first 3 nodes zookeeper url to nifi.zookeeper.connect.string. Once done I started nifi, however I get the below error. Posting the full stack trace, this stacktrace keeps on coming infinitely:

  1. 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
  2. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
  3. at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
  4. at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
  5. at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
  6. at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
  7. at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
  8. at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
  9. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  10. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  11. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  12. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  13. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  14. at java.lang.Thread.run(Thread.java:745)

Below is a longer stack-trace right from startup till the error:

  1. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures}
  2. 2018-11-05 06:09:41,961 INFO [main] org.eclipse.jetty.server.Server jetty-9.4.3.v20170317
  3. 2018-11-05 06:09:44,086 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1763ms
  4. 2018-11-05 06:09:44,247 INFO [main] org.eclipse.jetty.server.session DefaultSessionIdManager workerName=node0
  5. 2018-11-05 06:09:44,248 INFO [main] org.eclipse.jetty.server.session No SessionScavenger set, using defaults
  6. 2018-11-05 06:09:44,249 INFO [main] org.eclipse.jetty.server.session Scavenging every 660000ms
  7. 2018-11-05 06:09:44,293 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5b49b1df{/nifi-update-attribute-ui-1.7.1,file:///nifi/work/jetty/nifi-update-attribute-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-update-attribute-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-update-attribute-ui-1.7.1.war}
  8. 2018-11-05 06:09:44,915 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=511ms
  9. 2018-11-05 06:09:44,963 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@11ec2b2f{/nifi-standard-content-viewer-1.7.1,file:///nifi/work/jetty/nifi-standard-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-standard-content-viewer-1.7.1.war}
  10. 2018-11-05 06:09:45,009 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=34ms
  11. 2018-11-05 06:09:45,041 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@2446be5f{/nifi-image-viewer-1.7.1,file:///nifi/work/jetty/nifi-image-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-media-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-image-viewer-1.7.1.war}
  12. 2018-11-05 06:09:46,731 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1430ms
  13. 2018-11-05 06:09:46,770 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@71dfca65{/nifi-jolt-transform-json-ui-1.7.1,file:///nifi/work/jetty/nifi-jolt-transform-json-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-jolt-transform-json-ui-1.7.1.war}
  14. 2018-11-05 06:09:46,959 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=38ms
  15. 2018-11-05 06:09:46,965 INFO [main] org.eclipse.jetty.ContextHandler./nifi No Spring WebApplicationInitializer types detected on classpath
  16. 2018-11-05 06:09:47,006 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1d6dc2b8{/nifi,file:///nifi/work/jetty/nifi-web-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-ui-1.7.1.war}
  17. 2018-11-05 06:09:47,124 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=57ms
  18. 2018-11-05 06:09:47,164 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api No Spring WebApplicationInitializer types detected on classpath
  19. 2018-11-05 06:09:47,180 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api Initializing Spring root WebApplicationContext
  20. 2018-11-05 06:09:49,601 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties'
  21. 2018-11-05 06:09:49,602 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties'
  22. 2018-11-05 06:09:49,603 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 148 properties from /nifi/./conf/nifi.properties
  23. 2018-11-05 06:09:51,129 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE.Building now...
  24. 2018-11-05 06:09:51,437 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 94 properties from system properties and environment variables
  25. 2018-11-05 06:09:51,438 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 94 properties.Including precedence overrides effective accessible registry key size is 94
  26. 2018-11-05 06:09:51,485 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository using 256 partitions
  27. 2018-11-05 06:09:51,507 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 0 records
  28. 2018-11-05 06:09:51,527 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0
  29. 2018-11-05 06:09:51,533 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 5424567582 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size
  30. 2018-11-05 06:09:51,534 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false
  31. 2018-11-05 06:09:51,657 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates
  32. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures}
  33. 2018-11-05 06:09:51,671 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 checkpointed with 0 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1
  34. 2018-11-05 06:09:51,755 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected...
  35. 2018-11-05 06:09:51,826 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
  36. 2018-11-05 06:10:00,725 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected
  37. 2018-11-05 06:10:00,727 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
  38. 2018-11-05 06:10:01,832 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
  39. 2018-11-05 06:10:01,835 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election.
  40. 2018-11-05 06:10:01,836 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
  41. 2018-11-05 06:10:01,845 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election.
  42. 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started
  43. 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started
  44. 2018-11-05 06:10:04,629 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6149184e{/nifi-api,file:///nifi/work/jetty/nifi-web-api-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.7.1.war}
  45. 2018-11-05 06:10:05,255 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=463ms
  46. 2018-11-05 06:10:05,287 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath
  47. 2018-11-05 06:10:05,290 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@8eb1297{/nifi-content-viewer,file:///nifi/work/jetty/nifi-web-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.7.1.war}
  48. 2018-11-05 06:10:05,323 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=20ms
  49. 2018-11-05 06:10:05,325 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default
  50. 2018-11-05 06:10:05,367 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath
  51. 2018-11-05 06:10:05,370 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@646ba7be{/nifi-docs,file:///nifi/work/jetty/nifi-web-docs-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.7.1.war}
  52. 2018-11-05 06:10:05,421 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=35ms
  53. 2018-11-05 06:10:05,471 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath
  54. 2018-11-05 06:10:05,475 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1b6dab22{/,file:///nifi/work/jetty/nifi-web-error-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.7.1.war}
  55. 2018-11-05 06:10:05,492 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@2a654167{HTTP/1.1,[http/1.1]}{<node-ip>:8081}
  56. 2018-11-05 06:10:05,492 INFO [main] org.eclipse.jetty.server.Server Started @47964ms
  57. 2018-11-05 06:10:05,505 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...
  58. 2018-11-05 06:10:05,509 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 8082
  59. 2018-11-05 06:10:05,587 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
  60. 2018-11-05 06:10:05,610 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node:<node-ip>:8081
  61. 2018-11-05 06:10:13,959 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
  62. 2018-11-05 06:10:13,959 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
  63. 2018-11-05 06:10:21,670 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
  64. 2018-11-05 06:10:21,670 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
  65. 2018-11-05 06:10:21,675 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
  66. 2018-11-05 06:10:21,677 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1d2e0826 Connection State changed to SUSPENDED
  67. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures}
  68. 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
  69. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
  70. at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
  71. at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
  72. at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
  73. at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
  74. at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
  75. at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
  76. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  77. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  78. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  79. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  80. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  81. at java.lang.Thread.run(Thread.java:745)
  82. 2018-11-05 06:10:21,687 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
  83. org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
  84. at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838)
  85. at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
  86. at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
  87. at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
  88. at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  89. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  90. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  91. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  92. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  93. at java.lang.Thread.run(Thread.java:745)
4 REPLIES 4
Highlighted

Re: Unable to add a node to Nifi Cluster

Siddharth: I am getting same error. I am running nifi in container.Please let me know if you get the answer. Thanks!

Highlighted

Re: Unable to add a node to Nifi Cluster

Hi @Kamal Sehgal , Can you share some useful link about how did u setup the cluster because I am also configuring NiFi cluster with 3 machines (1 Manager and 2 Nodes) but could not be able to configure it yet. My NiFi version is 1.8.0 and using external Zookeeper instead of embedded Zookeeper.

Highlighted

Re: Unable to add a node to Nifi Cluster

Explorer

Hi @Kamal Sehgal:


What is the issue that you are facing.

Also I found this useful:

https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/

https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/


Its actually easier using an external zookeeper(first link has the steps) . In case you get stuck, let me know.

Also there is no concept of manager node in nifi now. Refer to, https://hortonworks.com/blog/apache-nifi-1-0-0-zero-master-clustering/

Highlighted

Re: Unable to add a node to Nifi Cluster

Hi @siddharth pande, I am also configuring NiFi cluster with 3 machines (1 Manager and 2 Nodes) but could not be able to configure it yet. Can you share some useful link in this regard. My NiFi version is 1.8.0 and using external Zookeeper instead of embedded Zookeeper.

Don't have an account?
Coming from Hortonworks? Activate your account here