In the final installment of Parameterize All The Things mini-series for the Cloudera Flow Management / Apache NiFi Best Practices cookbook, I will discuss Parameter Providers and how to use them.
As we've seen in the previous two posts, we can create Parameter Contexts and manually add key/value pairs to be used as Parameter references in NiFi component properties. But manually maintaining these values can be tedious and this doesn't lend itself to automation. Not only that, but it is possible that the parameter values should not be known by the Flow Designers and instead just used to connect to external systems, such as can be the case for database user passwords, secret keys, etc.
To solve this problem, the concept of Parameter Providers was introduced. This capability allows for parameters and their values (both sensitive and plain) to be fetched from an external source. These sources include (but aren't limited to) the following:
- Environment variables
- Database tables
- Password vaults such as Hashicorp, and CyberArk
- Secrets providers such as AWS, GCP, Azure, Kubernetes, and OnePassword
- A Java properties file (CFM 4.11+ only, not in Apache NiFi 2.x)
Parameter providers are "management controller services", which are global to the CFM/NiFi instance and are created and configured from the hamburger menu at the top-right of the NiFi UI.
For this post I will illustrate how to create, configure, and fetch parameters from an EnvironmentVariableParameterProvider. On the Parameter Providers tab (from the graphic above) I click the + icon on the far right to create a new instance (bringing up the dialog in the above graphic) and select EnvironmentVariableParameterProvider. This creates a new instance, and configuring it brings up the following dialog:
These are the default settings, to get all environment variables available to the Java Virtual Machine (JVM) running CFM/NiFi, and names the Parameter Group "Environment Variables". You can choose your own name as well as configure which environment variables you want to create parameters from, using a comma-separated list or a regular expression. For this example I will keep the defaults and click Apply.
Next I select the sideways-ellipsis icon to the right of the Parameter Provider, which offers me the choice to Fetch Parameters:
Choosing this brings up the following dialog:
From here you can choose to create and name a Parameter Context by clicking the checkbox in the middle, then selecting which of the parameters should be marked as sensitive:
For this example I will only mark my CLAUDE_KEY as sensitive, the rest will be non-sensitive parameters:
Clicking Apply then Close, I go to Parameter Contexts from the hamburger menu and Edit my Parameter Context:
You can see here that all my environment variables were imported as parameters and the CLAUDE_KEY parameter was marked as sensitive. This allows you to use fetched sensitive values in sensitive properties in CFM/NiFi components.
To tie all three posts in this series together, I set my Environment Variables parameter context as the parent of the DBCPConnectionPool Best Practice parameter context from part 2:
I hope this mini-series on Parameters in Cloudera Flow Management has been informative and helpful. Please leave any questions or found issues in the comments!
