Nifi on 3-node Docker cluster
I am using nifi in production environment and I need to run it in a 3-node cluster on docker (3 separate VMs).
Have used the following environment variables for Nifi:
version: '3.2'
services:
zookeeper:
hostname: zookeeper
image: bitnami/zookeeper:latest
container_name: collections_zookeeper
ports:
- "22181:2181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
nifi:
build:
context: ./nifi
dockerfile: Dockerfile
image: apache/nifi:custom
ports:
- 8080
environment:
- NIFI_WEB_HTTP_PORT=8080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_ADDRESS=1
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
- NIFI_ZK_CONNECT_STRING=zookeeper:22181
- NIFI_ELECTION_MAX_WAIT=1 min
Through this method, I am able to scale up nifi on a single VM, but need to run multiple instances of this on other nodes and run the same in the same cluster.
When I use the property:
NIFI_ZK_CONNECT_STRING=<ip_1>:2181,<ip_2:2181>,<ip_3:2181>
it renders the following stack:
2021-06-17 07:57:02,424 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2021-06-17 07:57:02,424 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
Any way to run nifi, zookeeper in 3 separate VMs in a cluster?
Thanks in advance!
