Solved: Re: Parameterized NiFi template handling - Cloudera Community - 199478

Support Questions

Find answers, ask questions, and share your expertise

Parameterized NiFi template handling

avatar

The system we are building will be multi-tenant and involve several NiFi Flows. To make this manageable we want the ability to define a single NiFi flow as a parameterized template, of which multiple instances can be started and stopped from the command-line with varying parameters.

One example flow might be a file pump: It consumes files of records from a watched folder, converts the contents to a matching Avro schema, writes to a Kafka topic for that record type. Having multiple record types, we need several instances of this flow running with the only differences being source folder, Avro schema name & Kafka topic name, all of which are simple parameters.

Similarly, the same NiFi flows will be used across the many tenants in the system, but started under different credentials to ensure there’s no possibility of ‘data crossing’ between tenants. Parameters might include the tenant ID, possibly derived from the credentials.

Requires

  • ability to define a NiFi template containing parameters
  • start any number of instances of that template with defined parameters from command-line
  • monitor running of the resulting flows. Note: Such flows should be read-only in the UI as it's too easy to mistakenly edit and break them.
  • stop a running instance from command-line

I can’t find anything in NiFi like these capabilities. Yes, there are templates and process groups, but manually building these via the UI is arduous and prone to error, and there’s no obvious way to generalize a flow (analogous to a class) and create it multiple times with different parameters (instance of a class).

The nearest I saw was this, from Andrew Grande of Hortonworks: https://github.com/aperepel/nifi-api-deploy

But it looks unofficial and not maintained (“Note: this was originally created for NiFi 0.x. REST APIs and concepts have changed significantly in NiFi 1.x”)

Is there anything more suitable or official out there? Or will we need to build a tool akin to (or starting from) Andrew’s, to drive the REST API?

Thanks,

Dave.

1 ACCEPTED SOLUTION

avatar

Hi @David Doran,

I am not sure there out of the box solution to your problem but there several assets you can use to implement this.

NiFi 1.5/HDF 3.1 introduced Variable Registry to handle this use case : https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

https://community.hortonworks.com/articles/154951/introduce-a-ui-driven-variable-registry-starting-f...

You have also a new flow registry that can be use as a repository to deploy these flows. Here an article that shows how to combine these two features : https://medium.com/@abdelkrim.hadjidj/fdlc-towards-flow-development-life-cycle-with-nifi-registries-...

Finally look to NiPyAPI for scripting these actions : https://github.com/Chaffelson/nipyapi

https://community.hortonworks.com/articles/167364/nifi-sdlc-automation-in-python-with-nipyapi-part-1...

Thanks

View solution in original post

3 REPLIES 3

avatar

Hi @David Doran,

I am not sure there out of the box solution to your problem but there several assets you can use to implement this.

NiFi 1.5/HDF 3.1 introduced Variable Registry to handle this use case : https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

https://community.hortonworks.com/articles/154951/introduce-a-ui-driven-variable-registry-starting-f...

You have also a new flow registry that can be use as a repository to deploy these flows. Here an article that shows how to combine these two features : https://medium.com/@abdelkrim.hadjidj/fdlc-towards-flow-development-life-cycle-with-nifi-registries-...

Finally look to NiPyAPI for scripting these actions : https://github.com/Chaffelson/nipyapi

https://community.hortonworks.com/articles/167364/nifi-sdlc-automation-in-python-with-nipyapi-part-1...

Thanks

avatar

Hi @Abdelkrim Hadjidj, thanks for your answer. I've begun looking onto your suggestions and I think we can probably build something suitable using a combination of the features you mentioned. Next week I’ll try experimenting with this. Thanks again. Dave.

avatar

Thanks @David Doran

If you found that this answer addressed your question, please take a moment to click "Accept" below.