Hi , Nifi is not able to form a cluster after restart , this was working fine for months without any issues, only after restart i am seeing issues , can anyone please help ? i have below zookeeper configuration and nifi.properties and nifi-app.log information. we are using external zookeeper.   zoo.cfg   initLimit=100 autopurge.purgeInterval=24 autopurge.snapRetainCount=30 syncLimit=50 tickTime=2000 dataDir=/ngs/app/xxx/zookeeper-data-356_1 admin.enableServer=true admin.serverPort=9990 standaloneEnabled=false server.1=node1:2887:3887;2184 server.2=node2:2886:3886;2182 server.3=node3:2889:3889;2183 quorum.cnxn.threads.size=20 4lw.commands.whitelist=mntr,stat   nifi.properties   nifi.state.management.configuration.file=./conf/state-management.xml # The ID of the local state provider nifi.state.management.provider.local=local-provider nifi.state.management.provider.cluster=zk-provider nifi.state.management.embedded.zookeeper.start=false nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties nifi.cluster.is.node=true nifi.cluster.node.address=node-xxxx nifi.cluster.node.protocol.port=11443 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=10 sec nifi.cluster.node.read.timeout=10 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=1 mins nifi.cluster.flow.election.max.candidates= nifi.zookeeper.connect.string=xxxx:2182,xxxx:2183,xxxx:2184 nifi.zookeeper.connect.timeout=3 secs nifi.zookeeper.session.timeout=3 secs nifi.zookeeper.root.node=/nifi     Nifi-app.log on node 1:   2022-01-09 05:55:56,151 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2022-01-09 05:55:56,379 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2022-01-09 05:55:56,521 INFO [main] org.apache.curator.utils.Compatibility Using emulated InjectSessionExpiration 2022-01-09 05:55:56,590 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2022-01-09 05:55:56,609 INFO [main] org.apache.zookeeper.common.X509Util Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2022-01-09 05:55:56,622 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2022-01-09 05:55:56,668 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2022-01-09 05:55:56,719 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2022-01-09 05:55:56,757 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.2=xxxxxx:2886:3886:participant;0.0.0.0:2182, server.1=xxxxxx:2887:3887:participant;0.0.0.0:2184, server.3=xxxxxx:2889:3889:participant;0.0.0.0:2183, version=0} 2022-01-09 05:55:56,777 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.2=xxxxxx:2886:3886:participant;0.0.0.0:2182, server.1=xxxxxx:2887:3887:participant;0.0.0.0:2184, server.3=xxxxxx:2889:3889:participant;0.0.0.0:2183, version=0} 2022-01-09 05:55:56,790 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2022-01-09 05:55:56,913 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. 2022-01-09 05:55:56,914 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2022-01-09 05:55:56,917 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2022-01-09 05:55:56,918 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2022-01-09 05:55:56,920 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2022-01-09 05:55:56,924 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2022-01-09 05:55:56,924 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2022-01-09 05:55:56,924 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2022-01-09 05:55:56,952 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED ..... 2022-01-09 05:56:15,810 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow 2022-01-09 05:56:18,732 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: rn-ssenpcit-lapp205.rno.apple.com:8086 2022-01-09 05:56:18,771 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-09 05:56:18,771 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-09 05:56:19,787 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-09 05:56:19,787 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-09 05:56:20,798 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again   Nifi-app.log on node 2 :    2022-01-09 05:55:40,827 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2022-01-09 05:55:40,893 INFO [main] org.apache.curator.utils.Compatibility Using emulated InjectSessionExpiration 2022-01-09 05:55:40,938 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2022-01-09 05:55:40,950 INFO [main] org.apache.zookeeper.common.X509Util Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2022-01-09 05:55:40,957 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2022-01-09 05:55:40,985 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2022-01-09 05:55:41,031 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2022-01-09 05:55:41,057 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.2=rn-ssenpcit-lapp206.rno.apple.com:2886:3886:participant;0.0.0.0:2182, server.1=rn-ssenpcit-lapp206.rno.apple.com:2887:3887:participant;0.0.0.0:2184, server.3=rn-ssenpcit-lapp206.rno.apple.com:2889:3889:participant;0.0.0.0:2183, version=0} 2022-01-09 05:55:41,067 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.2=rn-ssenpcit-lapp206.rno.apple.com:2886:3886:participant;0.0.0.0:2182, server.1=rn-ssenpcit-lapp206.rno.apple.com:2887:3887:participant;0.0.0.0:2184, server.3=rn-ssenpcit-lapp206.rno.apple.com:2889:3889:participant;0.0.0.0:2183, version=0} 2022-01-09 05:55:41,069 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2022-01-09 05:55:41,186 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. 2022-01-09 05:55:41,187 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2022-01-09 05:55:41,191 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2022-01-09 05:55:41,192 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2022-01-09 05:55:41,195 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2022-01-09 05:55:41,201 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2022-01-09 05:55:41,202 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2022-01-09 05:55:41,202 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2022-01-09 05:55:41,205 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED ...... 2022-01-09 05:55:57,209 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow 2022-01-09 05:56:00,250 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: rn-ssenpcit-lapp206.rno.apple.com:8086 2022-01-09 05:56:00,251 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-09 05:56:00,251 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-09 05:56:01,252 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-09 05:56:01,252 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-09     ... View more

I am seeing the below error, just after restarting NIFI, nothing is changed on nifi side, and on the external zookeeper side, it was working fine . Any help would be appreciated.i do not see any errors either in NIFI or ZooKeeper logs , just WARN messages Logs for NIFI:        2022-01-07 19:26:27,266 ERROR [main] org.apache.nifi.encrypt.StringEncryptor ******************************************************************************** 2022-01-07 19:26:27,267 ERROR [main] org.apache.nifi.encrypt.StringEncryptor * A blank sensitive properties key was provided * 2022-01-07 19:26:27,267 ERROR [main] org.apache.nifi.encrypt.StringEncryptor * Specify a unique key in nifi.properties * 2022-01-07 19:26:27,267 ERROR [main] org.apache.nifi.encrypt.StringEncryptor * for nifi.sensitive.props.key * 2022-01-07 19:26:27,267 ERROR [main] org.apache.nifi.encrypt.StringEncryptor * * 2022-01-07 19:26:27,267 ERROR [main] org.apache.nifi.encrypt.StringEncryptor * The Encrypt Config Tool in NiFi Toolkit can be used to * 2022-01-07 19:26:27,267 ERROR [main] org.apache.nifi.encrypt.StringEncryptor * migrate the flow to the new key * 2022-01-07 19:26:27,267 ERROR [main] org.apache.nifi.encrypt.StringEncryptor ******************************************************************************** 2022-01-07 19:26:27,505 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1938f3b3 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2022-01-07 19:26:27,506 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 1554 records in 26 milliseconds 2022-01-07 19:26:27,529 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1938f3b3 checkpointed with 1554 Records and 0 Swap Files in 23 milliseconds (Stop-the-world time = 3 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 24705695 2022-01-07 19:26:27,904 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {} to {rn-ssenpcit-lapp205.rno.apple.com:8086=NodeConnectionStatus[nodeId=rn-ssenpcit-lapp205.rno.apple.com:8086, state=DISCONNECTED, Disconnect Code=Has Not Yet Connected to Cluster, Disconnect Reason=Has Not Yet Connected to Cluster, updateId=1], xxxxxx:8086=NodeConnectionStatus[nodeId=xxxxxx:8086, state=DISCONNECTED, Disconnect Code=Has Not Yet Connected to Cluster, Disconnect Reason=Has Not Yet Connected to Cluster, updateId=0]} 2022-01-07 19:26:27,937 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 95 properties from system properties and environment variables 2022-01-07 19:26:27,938 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 95 properties. Including precedence overrides effective accessible registry key size is 95 2022-01-07 19:26:28,103 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository using 256 partitions 2022-01-07 19:26:28,180 INFO [main] o.a.n.p.store.WriteAheadStorePartition After recovering ./provenance_repository, next Event ID to be generated will be 843516108 2022-01-07 19:26:28,487 INFO [main] o.a.n.p.index.lucene.LuceneEventIndex Will avoid re-indexing Provenance Events because the newest index is defunct, so it will be re-indexed in the background 2022-01-07 19:26:28,573 INFO [pool-22-thread-1] o.a.n.p.index.lucene.LuceneEventIndex Determined that Max Event ID indexed for Partition default is approximately 843516107 based on index ./provenance_repository/lucene-8-index-1640336765330 2022-01-07 19:26:28,580 INFO [pool-22-thread-1] o.a.n.p.store.WriteAheadStorePartition The last Provenance Event indexed for partition default is 843506107, but the last event written to partition has ID 843516107. Re-indexing up to the last 10000 events to ensure that the Event Index is accurate and up-to-date 2022-01-07 19:26:29,072 INFO [pool-22-thread-1] o.a.n.p.store.WriteAheadStorePartition Finished re-indexing 660 events across 4 files for ./provenance_repository in 0.491 seconds 2022-01-07 19:26:29,073 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 35101377658 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2022-01-07 19:26:29,074 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2022-01-07 19:26:29,678 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2022-01-07 19:26:29,741 INFO [main] org.apache.curator.utils.Compatibility Using emulated InjectSessionExpiration 2022-01-07 19:26:29,781 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2022-01-07 19:26:29,792 INFO [main] org.apache.zookeeper.common.X509Util Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2022-01-07 19:26:29,799 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2022-01-07 19:26:29,820 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2022-01-07 19:26:29,878 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2022-01-07 19:26:29,896 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.2=xxxxxx:2886:3886:participant;0.0.0.0:2182, server.1=xxxxxx:2887:3887:participant;0.0.0.0:2184, server.3=xxxxxx:2889:3889:participant;0.0.0.0:2183, version=0} 2022-01-07 19:26:29,905 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.2=xxxxxx:2886:3886:participant;0.0.0.0:2182, server.1=xxxxxx:2887:3887:participant;0.0.0.0:2184, server.3=xxxxxx:2889:3889:participant;0.0.0.0:2183, version=0} 2022-01-07 19:26:29,908 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2022-01-07 19:26:30,023 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. 2022-01-07 19:26:30,024 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2022-01-07 19:26:30,027 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2022-01-07 19:26:30,029 INFO [main] org.apache.zookeeper.ClientCnxnSocket jute.maxbuffer value is 4194304 Bytes 2022-01-07 19:26:30,030 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Default schema 2022-01-07 19:26:30,033 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2022-01-07 19:26:30,036 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2022-01-07 19:26:30,036 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2022-01-07 19:26:30,043 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2022-01-07 19:26:30,043 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@7ee83317 Connection State changed to CONNECTED 2022-01-07 19:26:30,049 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.2=xxxxxx:2886:3886:participant;0.0.0.0:2182, server.1=xxxxxx:2887:3887:participant;0.0.0.0:2184, server.3=xxxxxx:2889:3889:participant;0.0.0.0:2183, version=0} 2022-01-07 19:26:30,049 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.2=xxxxxx:2886:3886:participant;0.0.0.0:2182, server.1=xxxxxx:2887:3887:participant;0.0.0.0:2184, server.3=xxxxxx:2889:3889:participant;0.0.0.0:2183, version=0} 2022-01-07 19:26:32,812 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@4317850d{nifi-api,/nifi-api,file:///ngs/app/ssewiret/nifi/work/jetty/nifi-web-api-1.10.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.10.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.10.0.war} 2022-01-07 19:26:33,040 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=87ms 2022-01-07 19:26:33,042 INFO [main] o.e.j.s.h.C._nifi_content_viewer No Spring WebApplicationInitializer types detected on classpath 2022-01-07 19:26:33,086 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@7e4c72d6{nifi-content-viewer,/nifi-content-viewer,file:///ngs/app/ssewiret/nifi/work/jetty/nifi-web-content-viewer-1.10.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.10.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-content-viewer-1.10.0.war} 2022-01-07 19:26:33,111 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=10ms 2022-01-07 19:26:33,113 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default 2022-01-07 19:26:33,113 INFO [main] o.e.j.s.h.ContextHandler._nifi_docs No Spring WebApplicationInitializer types detected on classpath 2022-01-07 19:26:33,153 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6cd64b3f{nifi-docs,/nifi-docs,file:///ngs/app/ssewiret/nifi/work/jetty/nifi-web-docs-1.10.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.10.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-docs-1.10.0.war} 2022-01-07 19:26:33,182 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=13ms 2022-01-07 19:26:33,183 INFO [main] o.e.j.server.handler.ContextHandler._ No Spring WebApplicationInitializer types detected on classpath 2022-01-07 19:26:33,215 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@51b01550{nifi-error,/,file:///ngs/app/ssewiret/nifi/work/jetty/nifi-web-error-1.10.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.10.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.10.0.war} 2022-01-07 19:26:33,300 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@1d74b85d(xxxxxx,h=[xxxxxx],w=[]) for SslContextFactory@3b349ea9[provider=null,keyStore=file:///ngs/app/ssewiret/certs/SSL/keystore.jks,trustStore=file:///ngs/app/ssewiret/certs/SSL/truststore.jks] 2022-01-07 19:26:33,366 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@459f703f{SSL,[ssl, http/1.1]}{xxxxxx:8086} 2022-01-07 19:26:33,366 INFO [main] org.eclipse.jetty.server.Server Started @32474ms 2022-01-07 19:26:33,390 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow... 2022-01-07 19:26:33,401 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 11443 2022-01-07 19:26:36,891 INFO [main] o.a.nifi.controller.ExtensionBuilder Created Controller Service of type org.apache.nifi.ssl.StandardSSLContextService with identifier 3877f1ad-017c-1000-ffff-ffffea055f7d 2022-01-07 19:26:51,497 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:51,498 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:26:52,498 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:52,499 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:26:53,500 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:53,500 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:26:54,501 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:54,501 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:26:55,502 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:55,503 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:26:56,504 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:56,504 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:26:57,505 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:57,505 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:26:58,506 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:58,506 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:26:59,507 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2022-01-07 19:26:59,507 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:27:00,507 INFO [main] o.a.nifi.controller.StandardFlowService Flow controller will load local dataflow and suspend connection handshake until a cluster connection response is received. 2022-01-07 19:27:00,537 INFO [main] o.apache.nifi.controller.FlowController Cluster State changed from Not Clustered to Clustered 2022-01-07 19:27:00,538 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Primary Node; this node is an active participant in the election. 2022-01-07 19:27:00,538 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2022-01-07 19:27:00,558 INFO [main] o.a.n.wali.SequentialAccessWriteAheadLog Recovering records from Write-Ahead Log at ./flowfile_repository 2022-01-07 19:27:00,620 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected 2022-01-07 19:27:02,186 INFO [main] org.apache.nifi.wali.HashMapSnapshot org.apache.nifi.wali.HashMapSnapshot@ddb8f8 restored 46662 Records and 0 Swap Files from Snapshot, ending with Transaction ID 439547579 2022-01-07 19:27:02,188 INFO [main] o.a.n.wali.SequentialAccessWriteAheadLog Successfully recovered 46662 records and 0 swap files from Snapshot at ./flowfile_repository/checkpoint with Max Transaction ID of 439547579 in 1629 milliseconds. Now recovering records from 1 journal files 2022-01-07 19:27:02,195 INFO [main] o.a.nifi.wali.LengthDelimitedJournal Recovering records from journal ./flowfile_repository/journals/439547580.journal 2022-01-07 19:27:02,199 INFO [main] o.a.nifi.wali.LengthDelimitedJournal Successfully recovered 0 updates from journal ./flowfile_repository/journals/439547580.journal     Logs for External ZooKeeper   2022-01-07 01:26:42,252 - WARN  [WorkerSender[myid=1]:QuorumCnxManager@679] - Cannot open channel to 3 at election address xxxxxx/10.203.2.73:3889 java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399 at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242 at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:591) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:714) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.base/java.lang.Thread.run(Thread.java:834) 2022-01-07 01:26:42,252 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x2107a70598 (n.zxid), 0x7 (n.round), LOOKING (n.state), 1 (n.sid), 0x22 (n.peerEPoch), LOOKING (my state)0 (n.config version) 2022-01-07 01:26:42,453 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):QuorumPeer@1269] - FOLLOWING 2022-01-07 01:26:42,453 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):ZooKeeperServer@938] - minSessionTimeout set to 4000 2022-01-07 01:26:42,453 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):ZooKeeperServer@947] - maxSessionTimeout set to 40000 2022-01-07 01:26:42,453 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):ZooKeeperServer@166] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /ngs/app/ssewiret/zookeeper-data-356_1/version-2 snapdir /ngs/app/ssewiret/zookeeper-data-356_1/version-2 2022-01-07 01:26:42,453 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 0 MS 2022-01-07 01:26:42,454 - WARN  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):Learner@282] - Unexpected exception, tries=0, remaining init limit=200000, connecting to xxxxxx/10.203.2.73:2886 java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:591) at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:233) at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:262) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:77) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1271) 2022-01-07 01:26:43,486 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):Learner@391] - Getting a diff from the leader 0x2107a70598 2022-01-07 01:26:43,487 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):Learner@546] - Learner received NEWLEADER message 2022-01-07 01:26:43,500 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):Learner@529] - Learner received UPTODATE message 2022-01-07 01:26:43,501 - INFO  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):CommitProcessor@256] - Configuring CommitProcessor with 4 worker threads. 2022-01-07 01:26:46,867 - INFO  [xxxxxx/10.203.2.73:3887:QuorumCnxManager$Listener@918] - Received connection request 10.203.2.73:56175 2022-01-07 01:26:46,875 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x2107a70598 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x22 (n.peerEPoch), FOLLOWING (my state)0 (n.config version) 2022-01-07 01:26:46,880 - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x2107a70598 (n.zxid), 0x7 (n.round), LOOKING (n.state), 3 (n.sid), 0x22 (n.peerEPoch), FOLLOWING (my state)0 (n.config version) 2022-01-07 01:28:13,957 - WARN  [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2184)(secure=disabled):Follower@125] - Got zxid 0x2300000001 expected 0x1 2022-01-07 01:28:13,960 - INFO  [SyncThread:1:FileTxnLog@216] - Creating new log file: log.2300000001 2022-01-07 01:28:14,033 - INFO  [NIOWorkerThread-1:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x101ffaa1b440000 2022-01-07 01:28:20,013 - INFO  [NIOWorkerThread-4:QuorumZooKeeperServer@157] - Submitting global closeSession request for session 0x101ffaa1b440002 2022-01-07 01:31:16,676 - INFO  [Snapshot Thread:FileTxnSnapLog@384] - Snapshotting: 0x2300015d93 to /ngs/app/ssewiret/zookeeper-data-356_1/version-2/snapshot.2300015d93 2022-01-07 01:31:16,678 - INFO  [SyncThread:1:FileTxnLog@216] - Creating new log file: log.2300015d96 2022-01-07 01:33:57,488 - INFO  [SyncThread:1:FileTxnLog@216] - Creating new log file: log.230002b1bb   ... View more

I have downloaded Nifi on my laptop running Mac OS Catalina Public Beta 2 and not able to access the UI. It is up and running and able to access all the REST API's shown below , http://127.0.0.1:8080/nifi-api/flow/processor-types http://127.0.0.1:8080/nifi-api/flow/controller-service-types http://127.0.0.1:8080/nifi-api/flow/reporting-task-types http://127.0.0.1:8080/nifi-api/controller/config http://127.0.0.1:8080/nifi-api/resources http://127.0.0.1:8080/nifi-api/system-diagnostics However UI is not accessible , getting below error. Note this is very specific to Mac OS Catalina public beta. it is working fine in Mojave . Nothing stands out in any of the logs. HTTP ERROR 404 Problem accessing /nifi/. Reason: Not Found Powered by Jetty:// 9.4.11.v20180605 For testing purpose , i have downloaded Nifi-Registry and it is also having the same issue. ... View more