I’d like to be able to have instances of NiFi running behind an AWS load balancer such that, if there is a surge of inbound requests (e.g., through AWS API Gateway), then AWS will auto-launch EC2 instances that fire up NiFi with my particular processor topology.
I’m not asking for a step-by-step HOWTO guide, but could someone point me toward any literature / previous guides / etc. that might help me in this effort?
A NiFi cluster can scale up pretty easily, but scaling down involves a lot more steps.
A new instance can be added to an existing cluster by simply configuring the new NIFI instance with the same NiFi conf directory files. If the cluster is secured, you would need to provide a new keystore and copy the existing truststore from another node.
On startup, the new NiFi will connect with the cluster and pull-down the cluster's flow.xml.gz, user.xml, and authorizations.xml files. It will then build all its local repositories based on defined paths from the nifi.properties file.
Any Remote Process Groups pointing at this cluster will learn about this new node on ext update (default 30 seconds).
Scaling down is a more complicated process.
Since a node now has active data, that node would need to be disconnected form cluster. Then its ingest processors stopped. Once all data has finished processing through all its dataflows, it could be shutdown completely. The node could then be dropped completely from cluster which is another manual step.