This article is part of the Cloudera Flow Management / Apache NiFi Best Practices Cookbook series and will discuss Best Practices for using Parameters and Parameter Contexts in your flows.
History
First a little background about Parameters: In NiFi 1.x, the original solution for being able to configure various environments (development vs production, e.g.) was Variables. This was a global key/value store of variable names with values associated with them. These variable names could be used in Expression Language expressions as well as certain properties in processors looking specifically for a variable name.
This single, global list was not flexible enough to solve many of the configuration challenges of our users. To solve this, Parameters were introduced. They are meant as a replacement for Variables and via Parameter Contexts are much more flexible and powerful due to inheritance (which I will discuss later).
Parameters
In general, a parameter is referred to by the following syntax in properties and expressions:
#{My Parameter}
where the parameter is named "My Parameter", versus a variable expression for "my.variable" which is as follows:
${my.variable}
Note that the syntax for parameters uses the "#" symbol where the syntax for parameters shares the Expression Language syntax using the "$" symbol. Naming rules are also different; for example parameter names can have spaces where variables cannot. You can use a parameter value alongside an Expression Language expression, such as:
#{My Parameter}.${literal('test')}
or just
#{My Parameter}.test
If the value of "My Parameter" is "this.is.a" then the output in both cases is "this.is.a.test".
The real power of parameters comes from grouping them and creating a hierarchy of those groups. Parameter groups are called Parameter Contexts, and a Parameter Context can be set on a Process Group (including the root process group). For a Process Group with a given Parameter Context, this means anywhere in the Process Group you can refer to its Parameters in any of the components.
To create a Parameter Context, go to the hamburger menu in the top-right corner of the UI and select Parameter Contexts. You can then select the + button to add a new one. You give it a name:
Then on the Parameters tab you can select the + button to add a new parameter giving it a name and a value and specifying whether that value is sensitive (such as a password) or not:
Now that we have our Parameters in our Parameter Context, we can assign that to our example Process Group on the canvas on its configuration menu:
Note the checkbox to "Apply recursively". If this is not checked, only the components inside the Process Group itself have access to the Parameters in the context. If you want all child Process Groups to recursively have access to the Parameters "down the tree", check this box. Apply the changes and we're ready to use this Parameter Context!
Overriding Parameter Values
Now imagine inside one of the child Process Groups, we want a different value for Username and Password. If we have applied the parent Process Group's Parameter Context, we will be using the parent's values for Username and Password. To override, we need a new Parameter Context, let's call it "Override Dev Process Group for Child PG 1". We then add Username and Password parameters with the different values (I chose "jsmith" as the Username and "joe's password" for Password):
Once this is applied, you select the new Parameter Context to be used by the aforementioned Child Process Group 1:
Looking at Child Process Group 2, you can see it has the parent's Parameter Context:
This is how you override parameters inside a child Process Group, while allowing other child process groups to use the parent's parameter values.
Swapping Parameter Contexts
This "overriding" concept can be applied to the parent process group. For example, let's say we want to run the same flow in the Production environment. You could edit the parameter values, but then the flow won't run on the Development system. The correct way to do this is to create a new process context for the production environment, setting the parameters and their values (in this case "mburgess" remains the Username and "prod_p@ssword" for Password:
When we want the flow to run in the production environment, we only need to swap the process context from Dev to Prod on the parent Process Group:
In this way you can run the same flow in multiple environments.
Summary
In this article I presented the history of Variables and Parameters, then discussed Parameter Contexts as groups of parameters. Using the inheritance and overriding features, I illustrated how parameter values can be overridden in specific child Process Groups, and otherwise inherited from the parent Process Group.
In the next article, I will extend this concept even more. Not only can you use parameters in processor properties for configuration, but you can use parameters to reference Controller Services so you can swap and/or override actual Controller Service implementations!
