Support Questions
Find answers, ask questions, and share your expertise

Nifi on 3-node Docker cluster

New Contributor

I am using nifi in production environment and I need to run it in a 3-node cluster on docker (3 separate VMs).

Have used the following environment variables for Nifi:

 

version: '3.2'
  services:
    zookeeper:
    hostname: zookeeper
    image: bitnami/zookeeper:latest
    container_name: collections_zookeeper
    ports:
      - "22181:2181"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes


  nifi:
    build:
      context: ./nifi
      dockerfile: Dockerfile
    image: apache/nifi:custom
    ports:
      - 8080
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_ADDRESS=1
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zookeeper:22181
      - NIFI_ELECTION_MAX_WAIT=1 min

 

Through this method, I am able to scale up nifi on a single VM, but need to run multiple instances of this on other nodes and run the same in the same cluster.

 

When I use the property:
NIFI_ZK_CONNECT_STRING=<ip_1>:2181,<ip_2:2181>,<ip_3:2181>

it renders the following stack:


2021-06-17 07:57:02,424 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2021-06-17 07:57:02,424 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered

Any way to run nifi, zookeeper in 3 separate VMs in a cluster?

Thanks in advance!