Supporting Custom Properties for Expression Language in NiFi

[#1]

NiFi has previously supported the ability to refer to flow file attributes, system properties and environment properties within expression language (EL); however the community requested an enhancement to also support custom properties. This would give users even more flexibility either in processing, handling flow content, or even in flow configuration (e.g. referring to a custom property in EL for connection, server or service properties).

In NiFi versions 0.7 & 1.0.0 an enhancement was added to allow administrators to define custom property files on nodes within their cluster and configure NiFi with their location so those properties could be loaded and available within EL. A new field in the nifi.properties file ( nifi.variable.registry.properties) is available for an administrator to set the paths of one or more custom properties files for use by NiFi.

Figure 1 - Custom Properties reference in nifi.properties

Once the nifi.properties file is updated custom attributes can be used as needed. NOTE: custom properties should contain distinct property values in order to ensure they won’t be overridden by other property files or by existing environment, system or flow file attributes.

For demonstration I have a flow that demonstrates use of custom properties in EL with UpdateAttributes processor and the PutFile processor

Figure 2 - Test Flow Writing Custom Attribute Data

Figure 3 - UpdateAttribute Advanced Configuration

Figure 4 - PutFile Config Screen with Directory using Custom Property in Expression

The output of this flow saves attributes created from custom property values to a folder location that is also defined by a custom property.

This custom properties enhancement sets the stage for developing a richer Variable Registry that will provide even more flexibility in custom property management providing UI driven administration, variable scope management and more.

For testing the flow in the above example, a template and referenced properties are available here:

https://gist.github.com/YolandaMDavis/364307c1ab5fe89b2edcef5647180873

[#2]

Hi Yolanda,

Can you please let me know how to use custom properties in minifi ?

[#3]

I want to develop a custom processor, which can access custom.properties. how to write in java code ?

[#4]

I have tried this mechanism and it works really well when using parameters for processors. But when it comes to parametrising controller services, the new key value pairs or updated key value pairs reflect in the services only after we restart the NiFi service. Is there a workaround to this??

The reason to parameterise controller service is that in an enterprise say we have many Oracle databases. Should we be creating a new controller service for every database or have just one controller service and pass it values from an external properties file??