Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

We are try to use external zookeeper for nifi cluster,But we are face error

avatar
Explorer

We are using ubuntu:20.04 OS.

Any configuration missed out? Any solution for this issue.

Zookeeper cluster is working well.when we try to use zookeeper in nifi at the point of time nifi is giving error.

zookeeper configuration : same configuration is available in 3 servers 

tickTime=2000
dataDir=/opt/zookeeper/data
clientPort=2181
maxClientCnxns=60
initLimit=10
syncLimit=5
server.1=nifi-zoo:2888:3888
server.2=nifi-zoo-2:2888:3888
server.3=nifi-zoo-3:2888:3888

nifi configuration

nifi-config-1

# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
nifi.state.management.provider.local=local-provider
nifi.state.management.provider.cluster=zk-provider
nifi.state.management.embedded.zookeeper.start=false
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

# Site to Site properties
nifi.remote.input.host=nifi-zoo
nifi.remote.input.secure=false
nifi.remote.input.socket.port=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
nifi.web.http.host=nifi-zoo
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=400
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.should.send.server.version=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-zoo
nifi.cluster.node.protocol.port=12000
nifi.cluster.node.protocol.threads=20
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=60 sec
nifi.cluster.node.read.timeout=60 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=30 sec
nifi.cluster.flow.election.max.candidates=

# cluster load balancing properties #
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=nifi-zoo:2181,nifi-zoo-2:2181,nifi-zoo-3:2181
nifi.zookeeper.connect.timeout=60 secs
nifi.zookeeper.session.timeout=60 secs
nifi.zookeeper.root.node=/nifi


nifi-config-2


# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
nifi.state.management.provider.local=local-provider
nifi.state.management.provider.cluster=zk-provider
nifi.state.management.embedded.zookeeper.start=false
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties

# Site to Site properties
nifi.remote.input.host=nifi-zoo-2
nifi.remote.input.secure=false
nifi.remote.input.socket.port=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
nifi.web.http.host=nifi-zoo-2
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=400
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.should.send.server.version=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-zoo-2
nifi.cluster.node.protocol.port=12000
nifi.cluster.node.protocol.threads=20
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=60 sec
nifi.cluster.node.read.timeout=60 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=30 sec
nifi.cluster.flow.election.max.candidates=

# cluster load balancing properties #
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=nifi-zoo:2181,nifi-zoo-2:2181,nifi-zoo-3:2181
nifi.zookeeper.connect.timeout=60 secs
nifi.zookeeper.session.timeout=60 secs
nifi.zookeeper.root.node=/nifi

 

nifi-config-3

 

# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
nifi.state.management.provider.local=local-provider
nifi.state.management.provider.cluster=zk-provider
nifi.state.management.embedded.zookeeper.start=false
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties


# Site to Site properties
nifi.remote.input.host=nifi-zoo-3
nifi.remote.input.secure=false
nifi.remote.input.socket.port=
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
nifi.web.http.host=nifi-zoo-3
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=400
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.should.send.server.version=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-zoo-3
nifi.cluster.node.protocol.port=12000
nifi.cluster.node.protocol.threads=20
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=60 sec
nifi.cluster.node.read.timeout=60 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=30 sec
nifi.cluster.flow.election.max.candidates=

# cluster load balancing properties #
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=nifi-zoo:2181,nifi-zoo-2:2181,nifi-zoo-3:2181
nifi.zookeeper.connect.timeout=60 secs
nifi.zookeeper.session.timeout=60 secs
nifi.zookeeper.root.node=/nifi

 

 

NIFI ERROR:

 

2021-02-04 11:12:47,783 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED
2021-02-04 11:12:49,106 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
2021-02-04 11:12:49,107 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED
2021-02-04 11:12:49,107 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED
2021-02-04 11:12:49,209 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2021-02-04 11:12:49,209 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED
2021-02-04 11:12:49,209 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED
2021-02-04 11:12:49,506 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
2021-02-04 11:12:49,507 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED
2021-02-04 11:12:49,507 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED
2021-02-04 11:12:49,508 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2021-02-04 11:12:49,508 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED
2021-02-04 11:12:49,508 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED
2021-02-04 11:12:49,508 ERROR [main-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:647)
at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152)
at org.apache.curator.framework.imps.FindAndDeleteProtectedNodeInBackground$2.processResult(FindAndDeleteProtectedNodeInBackground.java:104)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:630)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2021-02-04 11:12:50,214 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
2021-02-04 11:12:50,214 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED
2021-02-04 11:12:50,214 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED
2021-02-04 11:12:50,316 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2021-02-04 11:12:50,316 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED
2021-02-04 11:12:50,316 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED
2021-02-04 11:12:50,434 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
2021-02-04 11:12:50,434 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED
2021-02-04 11:12:50,434 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED
2021-02-04 11:12:50,536 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2021-02-04 11:12:50,536 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED
2021-02-04 11:12:50,536 ERROR [main-EventThread] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:647)
at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152)
at org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:222)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:601)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2021-02-04 11:12:50,536 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED
2021-02-04 11:12:51,990 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: RECONNECTED
2021-02-04 11:12:51,990 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to RECONNECTED
2021-02-04 11:12:51,990 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to RECONNECTED
2021-02-04 11:12:51,991 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException:
Cannot send heartbeat because there is no Cluster Coordinator currently elected
2021-02-04 11:12:52,092 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2021-02-04 11:12:52,093 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5a0453e0 Connection State changed to SUSPENDED
2021-02-04 11:12:52,093 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@5e3bcc78 Connection State changed to SUSPENDED

5 REPLIES 5

avatar
Explorer

Please help on this

avatar
Contributor

I am also facing the same issue.

after using an external zookeeper for distributed nifi getting this suspended and connected error.

Please let me know if you find any solution.

avatar
Explorer

Did any one resolve or understand this issue? Same issue also with external ZK clusters. Ours appear to happen after a bit of working-uptime (in this case 2 days)

avatar
Contributor

I resolved this by using embedded zookeepers.was getting the same error in the external one. started distributed nifi service in 3 nodes with embedded zookeeper.

avatar
Explorer

Need to use apache-zookeeper  for external one. Once you use apache-zookeeper it will work.

https://zookeeper.apache.org/releases.html