NiFi cluster sandbox on Docker

[#1]

The NiFi community maintains a NiFi Docker image that comes very handy when you want to play with NiFi on your laptop. When you develop flows to run in a NiFi cluster with 2 or more nodes, though, there are things that you need to check and test differently from a single-node NiFi cluster.

Recently, @Arqui shared a docker-compose configuration in this forum question that creates a 2-node NiFi cluster on Docker. The compose configuration still had a few issues to be ironed, though, and there were some manual steps you still had to perform before launching the cluster, like creating the TLS certificates for it.

I found the idea interesting and spent some time working out the missing details. I also automated the TLS certificate creation and added a load-balancer to the compose configuration. Now, to use your own NiFi sanbox cluster, all you need to do is this:

1. Download the docker-compose.yml file (see repo link below)
2. Run "docker compose up -d"
3. Connect to your own local 2-node cluster on http://localhost:8443/nifi.

I've been pretty happy with the result and that's what I now use on my laptop when I want to test something out quickly.

I hope this can be useful for others too: https://github.com/asdaraujo/nifi-stuff/tree/main/nifi-docker-compose

Cheers,

André

[#2]

Thanks for this.

[#3]

Hi @araujo ,

Thanks for the uploaded docker-compose.yml.  I have tried the steps in the here, but the command "docker-compose up -d" failed on the proxy container due to the fact that ./nginx.conf should have been a file. Would you elaborate the volumes setting for the proxy?  The proxy container in question taken from your repo is excerpted below. 

  proxy:
    image: nginx:latest
    container_name: proxy
#    volumes:
#      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "8443:8443"
    networks:
      - nifi
    depends_on:
      - nifi0
      - nifi1

Secondly, I need your help because after I brought up all containers with two line commented out above I cannot access this two-nodes NiFi cluster with the curl command below

$ curl -v https://localhost:8443/nifi
* Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* error:0A000126:SSL routines::unexpected eof while reading
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

Thanks,

David