Nifi on 3-node Docker cluster

yashratan — Thu, 17 Jun 2021 13:55:50 GMT

I am using nifi in production environment and I need to run it in a 3-node cluster on docker (3 separate VMs).

Have used the following environment variables for Nifi:

version: '3.2'
services:
zookeeper:
hostname: zookeeper
image: bitnami/zookeeper:latest
container_name: collections_zookeeper
ports:
- "22181:2181"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes

nifi:
build:
context: ./nifi
dockerfile: Dockerfile
image: apache/nifi:custom
ports:
- 8080
environment:
- NIFI_WEB_HTTP_PORT=8080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_ADDRESS=1
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
- NIFI_ZK_CONNECT_STRING=zookeeper:22181
- NIFI_ELECTION_MAX_WAIT=1 min

Through this method, I am able to scale up nifi on a single VM, but need to run multiple instances of this on other nodes and run the same in the same cluster.

When I use the property:
NIFI_ZK_CONNECT_STRING=<ip_1>:2181,<ip_2:2181>,<ip_3:2181>

it renders the following stack:

2021-06-17 07:57:02,424 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2021-06-17 07:57:02,424 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered

Any way to run nifi, zookeeper in 3 separate VMs in a cluster?

Thanks in advance!

Re: Nifi on 3-node Docker cluster

GPol — Fri, 01 Oct 2021 08:22:04 GMT

Hello,

I am having the same issue. Did you solved it?

Thank you.

Re: Nifi on 3-node Docker cluster

Green_ — Sat, 02 Oct 2021 22:56:55 GMT

@yashratan

Is it possible your nifi is configured to run the embedded zookeeper despite your trying to connect to your own zookeepers? Check if the nifi.state.management.embedded.zookeeper.start property in your nifi.properties file is set to true.

Check if you are able to communicate with all your zookeepers from each of your nodes. This definitely seems like an issue communicating with your ZKs.

Re: Nifi on 3-node Docker cluster

samarsimha — Sat, 08 Jan 2022 20:13:28 GMT

I am also seeing the same issue, Any update on this issue ?

question Re: Nifi on 3-node Docker cluster in Support Questions

Nifi on 3-node Docker cluster

Re: Nifi on 3-node Docker cluster

Re: Nifi on 3-node Docker cluster

Re: Nifi on 3-node Docker cluster