question Re: Parameterized NiFi template handling in Archives of Support Questions (Read Only)

Parameterized NiFi template handling

david_a_doran — Thu, 01 Feb 2018 22:17:54 GMT

The system we are building will be multi-tenant and involve several NiFi Flows. To make this manageable we want the ability to define a single NiFi flow as a parameterized template, of which multiple instances can be started and stopped from the command-line with varying parameters.

One example flow might be a file pump: It consumes files of records from a watched folder, converts the contents to a matching Avro schema, writes to a Kafka topic for that record type. Having multiple record types, we need several instances of this flow running with the only differences being source folder, Avro schema name & Kafka topic name, all of which are simple parameters.

Similarly, the same NiFi flows will be used across the many tenants in the system, but started under different credentials to ensure there’s no possibility of ‘data crossing’ between tenants. Parameters might include the tenant ID, possibly derived from the credentials.

Requires

ability to define a NiFi template containing parameters
start any number of instances of that template with defined parameters from command-line
monitor running of the resulting flows. Note: Such flows should be read-only in the UI as it's too easy to mistakenly edit and break them.
stop a running instance from command-line

I can’t find anything in NiFi like these capabilities. Yes, there are templates and process groups, but manually building these via the UI is arduous and prone to error, and there’s no obvious way to generalize a flow (analogous to a class) and create it multiple times with different parameters (instance of a class).

The nearest I saw was this, from Andrew Grande of Hortonworks: https://github.com/aperepel/nifi-api-deploy

But it looks unofficial and not maintained (“Note: this was originally created for NiFi 0.x. REST APIs and concepts have changed significantly in NiFi 1.x”)

Is there anything more suitable or official out there? Or will we need to build a tool akin to (or starting from) Andrew’s, to drive the REST API?

Thanks,

Dave.

Re: Parameterized NiFi template handling

ahadjidj — Fri, 02 Feb 2018 00:11:08 GMT

Hi @David Doran,

I am not sure there out of the box solution to your problem but there several assets you can use to implement this.

NiFi 1.5/HDF 3.1 introduced Variable Registry to handle this use case : https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

https://community.hortonworks.com/articles/154951/introduce-a-ui-driven-variable-registry-starting-f.html

You have also a new flow registry that can be use as a repository to deploy these flows. Here an article that shows how to combine these two features : https://medium.com/@abdelkrim.hadjidj/fdlc-towards-flow-development-life-cycle-with-nifi-registries-82e1ee866fab

Finally look to NiPyAPI for scripting these actions : https://github.com/Chaffelson/nipyapi

https://community.hortonworks.com/articles/167364/nifi-sdlc-automation-in-python-with-nipyapi-part-1.html

Thanks

Re: Parameterized NiFi template handling

david_a_doran — Fri, 02 Feb 2018 23:31:46 GMT

Hi @Abdelkrim Hadjidj, thanks for your answer. I've begun looking onto your suggestions and I think we can probably build something suitable using a combination of the features you mentioned. Next week I’ll try experimenting with this. Thanks again. Dave.

Re: Parameterized NiFi template handling

ahadjidj — Sat, 03 Feb 2018 15:17:58 GMT

Thanks @David Doran

If you found that this answer addressed your question, please take a moment to click "Accept" below.