Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Rising Star

NiFi has previously supported the ability to refer to flow file attributes, system properties and environment properties within expression language (EL); however the community requested an enhancement to also support custom properties. This would give users even more flexibility either in processing, handling flow content, or even in flow configuration (e.g. referring to a custom property in EL for connection, server or service properties).

In NiFi versions 0.7 & 1.0.0 an enhancement was added to allow administrators to define custom property files on nodes within their cluster and configure NiFi with their location so those properties could be loaded and available within EL. A new field in the nifi.properties file ( nifi.variable.registry.properties) is available for an administrator to set the paths of one or more custom properties files for use by NiFi.

7785-nifiproperties-fig1.png

Figure 1 - Custom Properties reference in nifi.properties

Once the nifi.properties file is updated custom attributes can be used as needed. NOTE: custom properties should contain distinct property values in order to ensure they won’t be overridden by other property files or by existing environment, system or flow file attributes.

For demonstration I have a flow that demonstrates use of custom properties in EL with UpdateAttributes processor and the PutFile processor

7786-nififlow-fig2.png

Figure 2 - Test Flow Writing Custom Attribute Data

7787-updateattributes-fig3.png

Figure 3 - UpdateAttribute Advanced Configuration

7788-putfileconfig-fig4.png

Figure 4 - PutFile Config Screen with Directory using Custom Property in Expression

The output of this flow saves attributes created from custom property values to a folder location that is also defined by a custom property.

This custom properties enhancement sets the stage for developing a richer Variable Registry that will provide even more flexibility in custom property management providing UI driven administration, variable scope management and more.

For testing the flow in the above example, a template and referenced properties are available here:

https://gist.github.com/YolandaMDavis/364307c1ab5fe89b2edcef5647180873

12,037 Views
Comments

Hi Yolanda,

Can you please let me know how to use custom properties in minifi ?

New Contributor

I want to develop a custom processor, which can access custom.properties. how to write in java code ?

New Contributor

I have tried this mechanism and it works really well when using parameters for processors. But when it comes to parametrising controller services, the new key value pairs or updated key value pairs reflect in the services only after we restart the NiFi service. Is there a workaround to this??

The reason to parameterise controller service is that in an enterprise say we have many Oracle databases. Should we be creating a new controller service for every database or have just one controller service and pass it values from an external properties file??

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 09:51 AM
Updated by:
 
Contributors
Top Kudoed Authors