I am using nifi in production environment and I need to run it in a 3-node cluster on docker (3 separate VMs).
Have used the following environment variables for Nifi:
version: '3.2' services: zookeeper: hostname: zookeeper image: bitnami/zookeeper:latest container_name: collections_zookeeper ports: - "22181:2181" environment: - ALLOW_ANONYMOUS_LOGIN=yes
nifi: build: context: ./nifi dockerfile: Dockerfile image: apache/nifi:custom ports: - 8080 environment: - NIFI_WEB_HTTP_PORT=8080 - NIFI_CLUSTER_IS_NODE=true - NIFI_CLUSTER_NODE_ADDRESS=1 - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082 - NIFI_ZK_CONNECT_STRING=zookeeper:22181 - NIFI_ELECTION_MAX_WAIT=1 min
Through this method, I am able to scale up nifi on a single VM, but need to run multiple instances of this on other nodes and run the same in the same cluster.
When I use the property: NIFI_ZK_CONNECT_STRING=<ip_1>:2181,<ip_2:2181>,<ip_3:2181>
it renders the following stack:
2021-06-17 07:57:02,424 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2021-06-17 07:57:02,424 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
Any way to run nifi, zookeeper in 3 separate VMs in a cluster?
Thanks in advance!
... View more