The system we are building will be multi-tenant and involve several NiFi Flows. To make this manageable we want the ability to define a single NiFi flow as a parameterized template, of which multiple instances can be started and stopped from the command-line with varying parameters.
One example flow might be a file pump: It consumes files of records from a watched folder, converts the contents to a matching Avro schema, writes to a Kafka topic for that record type. Having multiple record types, we need several instances of this flow running with the only differences being source folder, Avro schema name & Kafka topic name, all of which are simple parameters.
Similarly, the same NiFi flows will be used across the many tenants in the system, but started under different credentials to ensure there’s no possibility of ‘data crossing’ between tenants. Parameters might include the tenant ID, possibly derived from the credentials.
Requires
- ability to define a NiFi template containing parameters
- start any number of instances of that template with defined parameters from command-line
- monitor running of the resulting flows. Note: Such flows should be read-only in the UI as it's too easy to mistakenly edit and break them.
- stop a running instance from command-line
I can’t find anything in NiFi like these capabilities. Yes, there are templates and process groups, but manually building these via the UI is arduous and prone to error, and there’s no obvious way to generalize a flow (analogous to a class) and create it multiple times with different parameters (instance of a class).
The nearest I saw was this, from Andrew Grande of Hortonworks: https://github.com/aperepel/nifi-api-deploy
But it looks unofficial and not maintained (“Note: this was originally created for NiFi 0.x. REST APIs and concepts have changed significantly in NiFi 1.x”)
Is there anything more suitable or official out there? Or will we need to build a tool akin to (or starting from) Andrew’s, to drive the REST API?
Thanks,
Dave.
