NiFi Registry export/import best practices?

[#1]

We are trying to implement best practices in regards export/import via NiFi Registry. Currently we have one DEV site and multiple operational sites, ideally we would export from DEV and import into our various OPS sites. With that in mind I have a few questions:

• What is the best way to provide site-specific configuration for the controllers/Processors?
  • Should we have a unique properties file for each site and we select which one to deploy per site?
    • According to some documentation, later versions of NiFi are planning to remove support for this kind of unique property file loading.
  • Is there a way in the controllers/Processors to provide conditional values based on deployed site?
    • Of particular concern is our Solr "Zookeeper Host List"
      • This is a property value on our Query and Ingest processors, and they currently aren't enabled to support the Variable Registry, so the value is explicitly provided (no variable lookup/replacement).
    • Do we need to set up and use a "Variable Registry" at each site?
      • Is there a way to import a set of variables or do we have to define the variables via the UI?
      • Documentation again suggests moving away from variables, and use parameters instead.
    • Should we use Parameters instead. What is the best way to manage/deploy site-specific set of Parameters?
• What is the best way to export from one NiFi Registry, import into another NiFi Registry instance, and then trigger the associated NiFi instance to update its workflow reference?
  • Does the update need to be triggered via the UI, or can the update be automated?

[#2 · Accepted solution]

@jennings0131 
Here are my thoughts based on your questions:

• What is the best way to provide site-specific configuration for the controllers/Processors?  <--- The recommended approach is to use Parameter contexts [1].  NiFi Parameters can be used in component properties that do not support NiFi Expression Language (EL).
  • Should we have a unique properties file for each site and we select which one to deploy per site?  <--- You would typically create a unique parameter context per each site (same parameter strings with unique values per site)
  • Is there a way in the controllers/Processors to provide conditional values based on deployed site?  <-- I am not clear on the ask in this question.
    • Of particular concern is our Solr "Zookeeper Host List"
      • This is a property value on our Query and Ingest processors, and they currently aren't enabled to support the Variable Registry, so the value is explicitly provided (no variable lookup/replacement).  <--- All properties support NiFi Parameters, so you convert these property's values into parameters.  
    • Do we need to set up and use a "Variable Registry" at each site?  <--- I would not use variable registry in your use case. 
      • Is there a way to import a set of variables or do we have to define the variables via the UI?  <-- Anything you can do via the UI, you can do via NiFi's rest-api [2] as well.

      • 

      • Documentation again suggests moving away from variables, and use parameters instead.
    • Should we use Parameters instead. What is the best way to manage/deploy site-specific set of Parameters? <-- Yes, This is what i would recommend as well.
• What is the best way to export from one NiFi Registry, import into another NiFi Registry instance, and then trigger the associated NiFi instance to update its workflow reference?  <-- You could certainly automated this via an external custom script.  The NiFi CLI toolkit [3] offers many functions for interacting with the NiFi-Registry allowing you to export flows and import flows.
  • Does the update need to be triggered via the UI, or can the update be automated?

One of the easiest ways when getting started with the NiFi rest-api is to make use of the developer tools in your browser.  With developer tools open you can perform the actions via the UI and observe the rest-api requests that are used.  Most developer tools even allow you to copy that request as a curl command which you could simply execute from a terminal window to produce same outcome:

Automation would utilize a combination of NiFi CLI toolkit and NiFi Rest-api commands.   Keep in mind that when you import a new flow version to another registry, NiFi will not know about it right away.  NiFi checks NiFi-Registry for changes via a background thread.

What makes Parameters great:
1. Any property can utilize parameters (even those that do not support NiFi Expression Language

2. Once you create a parameter context and it is associated with a Process Group, you can easily convert existing component property values into parameters using the "convert to parameter" to the right of the property,
3. On the other NiFi site, you would not need to modify the processor properties.  Simply associate a site specific parameter context to the PG that contains all the same parameters just with site specific values.
4. Even component password property values (sensitive encrypted properties) can be converted in to parameters.

[1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters

[2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

[3] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI

If you found this assisted with your query, please take a moment to login and click on "Accept as Solution" found below this post.

Thank you,

Matt

[#3]

@MattWho  Thank you for the thorough reply!

A couple things we will need to work out going forward with using Parameters:
1. a way to update site NiFi instances with any updates to the Parameter Contexts

2. a way to update the workflow via the Registry, but not change which Parameter Context the processor groups are assigned to.