Hi @DigitalPlumber ,   Just to give few more details to above my reply. Below is the version of the Nifi I am using for clustering, Also if you can see in the screenshot, this nifi is in clustered with below entries in configuration file, nifi.properties nifi.zookeeper.connect.string=192.168.0.10:2182 zookeeper.properties server.1=192.168.0.10:2777:3777;2182 state-management.xml <property name="Connect String">192.168.0.10:2182</property> Below below is the screenshot of the clustered node, These all screenshot which I have shared in reply in my both the comments are latest (like are of same day or day before I am commenting).    Also on top of it, can you please give me more clarity on zookeeper.properties file node entry. I want to know more about port range which we are providing for node entry,   zookeeper.properties server.1=192.168.0.10:2777:3777;2182     2777:3777  What is use of above port range value ? What is minimum and maximum value for this ? Is it necessary to have difference of 1000 between these ports ?  (3777 - 2777 = 1000) If we want to change the value of these port range then what is range of it ?   Thanks in advance. ... View more

Hi @DigitalPlumber ,   Thanks for the reply. As I mentioned earlier I am able to achieve Secure Nifi Clustering. But the problem is if first node is removed out of 3 nodes cluster is getting impacted badly with connection loss.   And I was trying one more solution like to have same zookeeper string across the cluster. So now in nifi.properties, zookeeper.properties and state-management.xml file having common entry compared to all other nodes. Like below,   nifi.properties   nifi.state.management.configuration.file=./conf/state-management.xml # The ID of the local state provider nifi.state.management.provider.local=local-provider # The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster. nifi.state.management.provider.cluster=zk-provider # Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server nifi.state.management.embedded.zookeeper.start=true # Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties # web properties # nifi.web.https.host=192.168.0.10 nifi.web.https.port=2520 # security properties # nifi.security.keystore=./ext-lib/ssl/keystore.jks nifi.security.keystoreType=jks nifi.security.keystorePasswd=CskME7zgbZiR1k/vwlPJcOayK4VPl3+gVIq/ZgD9c nifi.security.keyPasswd=CskME7zgbiR1k/vwlPJc3OayK4VP3+gVIq/ZgUD9c nifi.security.truststore=./ext-lib/ssl/truststore.jks nifi.security.truststoreType=jks nifi.security.truststorePasswd=dHNilK1qAKsX5ee3e7gcbg/yQQsVSbUrxGG4Lhr7Y nifi.security.needClientAuth=true nifi.security.user.authorizer=managed-authorizer nifi.security.user.login.identity.provider=file-identity-provider nifi.security.ocsp.responder.url= nifi.security.ocsp.responder.certificate= # cluster common properties (all nodes must have same values) # nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.is.secure=true # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=192.168.0.10 nifi.cluster.node.protocol.port=2510 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=1 mins nifi.cluster.flow.election.max.candidates=10 # cluster load balancing properties # nifi.cluster.load.balance.host= nifi.cluster.load.balance.port=6343 nifi.cluster.load.balance.connections.per.node=4 nifi.cluster.load.balance.max.thread.count=8 nifi.cluster.load.balance.comms.timeout=30 sec # zookeeper properties, used for cluster management # nifi.zookeeper.connect.string=192.168.0.10:2182,192.168.0.20:2182,192.168.0.30:2182 nifi.zookeeper.connect.timeout=3 secs nifi.zookeeper.session.timeout=3 secs nifi.zookeeper.root.node=/nifi     zookeeper.properties   initLimit=10 autopurge.purgeInterval=24 syncLimit=5 tickTime=2000 dataDir=./ext-lib/zookeeper-home/zookeeper autopurge.snapRetainCount=30 server.1=192.168.0.10:2777:3777;2182 server.2=192.168.0.20:2777:3777;2182 server.3=192.168.0.30:2777:3777;2182     state-management.xml   <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <stateManagement> <local-provider> <id>local-provider</id> <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class> <property name="Directory">./state/local</property> <property name="Always Sync">false</property> <property name="Partitions">16</property> <property name="Checkpoint Interval">2 mins</property> </local-provider> <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">192.168.0.10:2182,192.168.0.20:2182,192.168.0.30:2182</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </cluster-provider> </stateManagement>     Now after these configuration if I am moved ahead to start the server then I am getting below continuous errors.     2020-08-06 22:55:21,397 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-08-06 22:55:21,397 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-08-06 22:55:21,399 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)     Above are infinite logs and even it will start the Nifi. Once I am trying to access the nifi from the browser then I am getting cluster connection error message. Please see below screenshot for your reference.                               Its quite surprising for me. As If I am starting the server with in cluster with single zookeeper string, I am able to start without any issue. Single node is getting into cluster properly.   nifi.properties nifi.zookeeper.connect.string=192.168.0.10:2182 zookeeper.properties server.1=192.168.0.10:2777:3777;2182 state-management.xml <property name="Connect String">192.168.0.10:2182</property>     But When I am moving ahead to add more than one node it is failing.   nifi.properties nifi.zookeeper.connect.string=192.168.0.10:2182,192.168.0.20:2182,192.168.0.30:2182 zookeeper.properties server.1=192.168.0.10:2777:3777;2182 server.2=192.168.0.20:2777:3777;2182 server.3=192.168.0.30:2777:3777;2182 state-management.xml <property name="Connect String">192.168.0.10:2182,192.168.0.20:2182,192.168.0.30:2182</property>     I have followed all the steps for the same. If I missed out any of the step then even I won't be able to achieve 1 scenario. Please let me know if I am missing anything. If I am missing any of the configuration then of course it will be really very silly mistake which I am not able to figure it out for now. :).   Again thanks @DigitalPlumber  for taking out time and replying on this thread. Please let me know what solution will work for this.   Thanks in advance. ... View more