Supporting Custom Properties for Expression Language in NiFi
NiFi has previously supported the ability to refer to flow file attributes, system properties and environment properties within expression language (EL); however the community requested an enhancement to also support custom properties. This would give users even more flexibility either in processing, handling flow content, or even in flow configuration (e.g. referring to a custom property in EL for connection, server or service properties).
In NiFi versions 0.7 & 1.0.0 an enhancement was added to allow administrators to define custom property files on nodes within their cluster and configure NiFi with their location so those properties could be loaded and available within EL. A new field in the nifi.properties file ( nifi.variable.registry.properties) is available for an administrator to set the paths of one or more custom properties files for use by NiFi.
Figure 1 - Custom Properties reference in nifi.properties
Once the nifi.properties file is updated custom attributes can be used as needed. NOTE: custom properties should contain distinct property values in order to ensure they won’t be overridden by other property files or by existing environment, system or flow file attributes.
For demonstration I have a flow that demonstrates use of custom properties in EL with UpdateAttributes processor and the PutFile processor
Figure 2 - Test Flow Writing Custom Attribute Data
Figure 3 - UpdateAttribute Advanced Configuration
Figure 4 - PutFile Config Screen with Directory using Custom Property in Expression
The output of this flow saves attributes created from custom property values to a folder location that is also defined by a custom property.
This custom properties enhancement sets the stage for developing a richer Variable Registry that will provide even more flexibility in custom property management providing UI driven administration, variable scope management and more.
For testing the flow in the above example, a template and referenced properties are available here:
https://gist.github.com/YolandaMDavis/364307c1ab5fe89b2edcef5647180873
