Configuring the Git Persistence Provider for the NiFi Registry under CDF/CFM
OBJECTIVE:
Step-by step instructions for enabling the git persistence provider for NiFi in Cloudera Flow Management (CFM) on Cloudera Manager.
OVERVIEW:
The default install of early releases of Cloudera Flow Management (CFM) includes only basic configurations for NiFi Registry in Cloudera Manager.
This article explains how to reconfigure the Nifi Registry persistence providers to use git instead of the filesystem provider.
PREREQUISITES:
CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later
ADDING NiFi METRICS TO CM:
1. Pick a location where the Nifi Registry will store local git repository files. This location has to be accessible to the NiFi Registry service account (typically: nifiregistry). It must also be initialized as a git repository using git init and linked to a remote repository using git remote add origin. (Alternatively, an existing git repository can be cloned to create the folder using git clone). The substitution variable ${nifi.registry.working.directory} can be used to point to the install location for NiFi Registry (usually /var/lib/nifiregistry/ when NiFi registry is installed by Cloudera Manager).
2. From the NIFIREGISTRY Configuration tab in Cloudera Manager, search for “flowPersistenceProvider”
3. Locate the “NiFi Registry Advanced Configuration Snippet (Safety Valve) for staging/providers.xml” Safety Valve configuration.
4. Add the following Name/Value Pairs:
Name
| Value (example)
| Description
|
xml.providers.flowPersistenceProvider.git-provider.class | org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider | Provider java class for git - should always be org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider |
xml.providers.flowPersistenceProvider.git-provider.property.Flow Storage Directory | ${nifi.registry.working.directory}/git/flow_storage | Local directory for git project. Must be git initialized and have a remote defined. Use https link |
xml.providers.flowPersistenceProvider.git-provider.property.Remote To Push | origin | Name of remote to push to - typically "origin" |
xml.providers.flowPersistenceProvider.git-provider.property.Remote Access User | somegitusername | User account for git remote repository |
xml.providers.flowPersistenceProvider.git-provider.property.Remote Access Password | 1234mygitaccesstoken567 | Personal access token for Github API |
Sample XML
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote Access Password</name>
<value>1234mygitaccesstoken567</value>
<description>Personal access token for Github API</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.class</name>
<value>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</value>
<description>Provider java class for git - should always be org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Flow Storage Directory</name>
<value>${nifi.registry.working.directory}/git/flow_storage</value>
<description>Local directory for git project. Must be git initialized and have a remote defined. Use https link</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote To Push</name>
<value>origin</value>
<description>Name of remote to push to - typically "origin"</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote Access User</name>
<value>somegitusername</value>
<description>User account for git remote repository</description>
</property>
5. From the NIFIREGISTRY Configuration tab in Cloudera Manager, search for “providers ignored properties”
6. Locate the “providers ignored properties” (nifi.registry.providers.ignored) configuration.
7. Add the following two properties to the ignored list:
xml.providers.flowPersistenceProvider.file-provider.property.Flow Storage Directoryxml.providers.flowPersistenceProvider.file-provider.class
(this prevents NiFi Registry from adding the default filesystem provider to the providers.xml along with the git provider, which would be invalid)
CONCLUSION:
After making the changes above and restarting the NiFi Registry service, you should be able to enable a registry client in the NiFi Controller Settings, then start version control in the NiFi canvas and see your changes reflected in the remote git repository (such as GitHub). For more information about versioning and dev/ops workflows using the NiFi Flow Registry with git, check out Pierre Villard’s excellent article here.
