Hi,
I have a running cluster with 3 nifi nodes, all running an embedded zookeeper. Now I want to add a fourth node which doesn't run an embedded zookeeper. What I did was, on the new node edited the nifi properties, changed nifi.cluster.is.node to true, and added the first 3 nodes zookeeper url to nifi.zookeeper.connect.string. Once done I started nifi, however I get the below error. Posting the full stack trace, this stacktrace keeps on coming infinitely:
- 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
- org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
- at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
- at java.util.concurrent.FutureTask.run(FutureTask.java:266)
- at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
- at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
- at java.lang.Thread.run(Thread.java:745)
Below is a longer stack-trace right from startup till the error:
- p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
- 2018-11-05 06:09:41,961 INFO [main] org.eclipse.jetty.server.Server jetty-9.4.3.v20170317
- 2018-11-05 06:09:44,086 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1763ms
- 2018-11-05 06:09:44,247 INFO [main] org.eclipse.jetty.server.session DefaultSessionIdManager workerName=node0
- 2018-11-05 06:09:44,248 INFO [main] org.eclipse.jetty.server.session No SessionScavenger set, using defaults
- 2018-11-05 06:09:44,249 INFO [main] org.eclipse.jetty.server.session Scavenging every 660000ms
- 2018-11-05 06:09:44,293 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5b49b1df{/nifi-update-attribute-ui-1.7.1,file:///nifi/work/jetty/nifi-update-attribute-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-update-attribute-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-update-attribute-ui-1.7.1.war}
- 2018-11-05 06:09:44,915 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=511ms
- 2018-11-05 06:09:44,963 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@11ec2b2f{/nifi-standard-content-viewer-1.7.1,file:///nifi/work/jetty/nifi-standard-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-standard-content-viewer-1.7.1.war}
- 2018-11-05 06:09:45,009 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=34ms
- 2018-11-05 06:09:45,041 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@2446be5f{/nifi-image-viewer-1.7.1,file:///nifi/work/jetty/nifi-image-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-media-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-image-viewer-1.7.1.war}
- 2018-11-05 06:09:46,731 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1430ms
- 2018-11-05 06:09:46,770 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@71dfca65{/nifi-jolt-transform-json-ui-1.7.1,file:///nifi/work/jetty/nifi-jolt-transform-json-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-jolt-transform-json-ui-1.7.1.war}
- 2018-11-05 06:09:46,959 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=38ms
- 2018-11-05 06:09:46,965 INFO [main] org.eclipse.jetty.ContextHandler./nifi No Spring WebApplicationInitializer types detected on classpath
- 2018-11-05 06:09:47,006 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1d6dc2b8{/nifi,file:///nifi/work/jetty/nifi-web-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-ui-1.7.1.war}
- 2018-11-05 06:09:47,124 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=57ms
- 2018-11-05 06:09:47,164 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api No Spring WebApplicationInitializer types detected on classpath
- 2018-11-05 06:09:47,180 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api Initializing Spring root WebApplicationContext
- 2018-11-05 06:09:49,601 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties'
- 2018-11-05 06:09:49,602 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties'
- 2018-11-05 06:09:49,603 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 148 properties from /nifi/./conf/nifi.properties
- 2018-11-05 06:09:51,129 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE.Building now...
- 2018-11-05 06:09:51,437 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 94 properties from system properties and environment variables
- 2018-11-05 06:09:51,438 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 94 properties.Including precedence overrides effective accessible registry key size is 94
- 2018-11-05 06:09:51,485 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository using 256 partitions
- 2018-11-05 06:09:51,507 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 0 records
- 2018-11-05 06:09:51,527 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0
- 2018-11-05 06:09:51,533 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 5424567582 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size
- 2018-11-05 06:09:51,534 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false
- 2018-11-05 06:09:51,657 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates
- p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
- 2018-11-05 06:09:51,671 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 checkpointed with 0 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1
- 2018-11-05 06:09:51,755 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected...
- 2018-11-05 06:09:51,826 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
- 2018-11-05 06:10:00,725 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected
- 2018-11-05 06:10:00,727 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
- 2018-11-05 06:10:01,832 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
- 2018-11-05 06:10:01,835 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election.
- 2018-11-05 06:10:01,836 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
- 2018-11-05 06:10:01,845 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election.
- 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started
- 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started
- 2018-11-05 06:10:04,629 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6149184e{/nifi-api,file:///nifi/work/jetty/nifi-web-api-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.7.1.war}
- 2018-11-05 06:10:05,255 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=463ms
- 2018-11-05 06:10:05,287 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath
- 2018-11-05 06:10:05,290 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@8eb1297{/nifi-content-viewer,file:///nifi/work/jetty/nifi-web-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.7.1.war}
- 2018-11-05 06:10:05,323 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=20ms
- 2018-11-05 06:10:05,325 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default
- 2018-11-05 06:10:05,367 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath
- 2018-11-05 06:10:05,370 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@646ba7be{/nifi-docs,file:///nifi/work/jetty/nifi-web-docs-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.7.1.war}
- 2018-11-05 06:10:05,421 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=35ms
- 2018-11-05 06:10:05,471 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath
- 2018-11-05 06:10:05,475 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1b6dab22{/,file:///nifi/work/jetty/nifi-web-error-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.7.1.war}
- 2018-11-05 06:10:05,492 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@2a654167{HTTP/1.1,[http/1.1]}{<node-ip>:8081}
- 2018-11-05 06:10:05,492 INFO [main] org.eclipse.jetty.server.Server Started @47964ms
- 2018-11-05 06:10:05,505 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...
- 2018-11-05 06:10:05,509 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 8082
- 2018-11-05 06:10:05,587 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
- 2018-11-05 06:10:05,610 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node:<node-ip>:8081
- 2018-11-05 06:10:13,959 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
- 2018-11-05 06:10:13,959 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
- 2018-11-05 06:10:21,670 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
- 2018-11-05 06:10:21,670 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
- 2018-11-05 06:10:21,675 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
- 2018-11-05 06:10:21,677 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1d2e0826 Connection State changed to SUSPENDED
- p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}
- 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
- org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
- at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
- at java.util.concurrent.FutureTask.run(FutureTask.java:266)
- at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
- at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
- at java.lang.Thread.run(Thread.java:745)
- 2018-11-05 06:10:21,687 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
- org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
- at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
- at java.util.concurrent.FutureTask.run(FutureTask.java:266)
- at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
- at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
- at java.lang.Thread.run(Thread.java:745)