Configuring the Git Persistence Provider for the NiFi Registry under CDF/CFM

[#1]

OBJECTIVE:

Step-by step instructions for enabling the git persistence provider for NiFi in Cloudera Flow Management (CFM) on Cloudera Manager.

OVERVIEW:

The default install of early releases of Cloudera Flow Management (CFM) includes only basic configurations for NiFi Registry in Cloudera Manager.

This article explains how to reconfigure the Nifi Registry persistence providers to use git instead of the filesystem provider.

PREREQUISITES:

CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later

ADDING NiFi METRICS TO CM:

1. Pick a location where the Nifi Registry will store local git repository files. This location has to be accessible to the NiFi Registry service account (typically: nifiregistry). It must also be initialized as a git repository using git init and linked to a remote repository using git remote add origin. (Alternatively, an existing git repository can be cloned to create the folder using git clone). The substitution variable ${nifi.registry.working.directory} can be used to point to the install location for NiFi Registry (usually /var/lib/nifiregistry/ when NiFi registry is installed by Cloudera Manager).

2. From the NIFIREGISTRY Configuration tab in Cloudera Manager, search for “flowPersistenceProvider”

3. Locate the “NiFi Registry Advanced Configuration Snippet (Safety Valve) for staging/providers.xml” Safety Valve configuration.

4. Add the following Name/Value Pairs:

Name	Value (example)	Description
xml.providers.flowPersistenceProvider.git-provider.class	org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider	Provider java class for git - should always be org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider
xml.providers.flowPersistenceProvider.git-provider.property.Flow Storage Directory	${nifi.registry.working.directory}/git/flow_storage	Local directory for git project. Must be git initialized and have a remote defined. Use https link
xml.providers.flowPersistenceProvider.git-provider.property.Remote To Push	origin	Name of remote to push to - typically "origin"
xml.providers.flowPersistenceProvider.git-provider.property.Remote Access User	somegitusername	User account for git remote repository
xml.providers.flowPersistenceProvider.git-provider.property.Remote Access Password	1234mygitaccesstoken567	Personal access token for Github API

Sample XML

<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote Access Password</name>
<value>1234mygitaccesstoken567</value>
<description>Personal access token for Github API</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.class</name>
<value>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</value>
<description>Provider java class for git - should always be org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Flow Storage Directory</name>
<value>${nifi.registry.working.directory}/git/flow_storage</value>
<description>Local directory for git project. Must be git initialized and have a remote defined. Use https link</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote To Push</name>
<value>origin</value>
<description>Name of remote to push to - typically "origin"</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote Access User</name>
<value>somegitusername</value>
<description>User account for git remote repository</description>
</property>

5. From the NIFIREGISTRY Configuration tab in Cloudera Manager, search for “providers ignored properties”

6. Locate the “providers ignored properties” (nifi.registry.providers.ignored) configuration.

7. Add the following two properties to the ignored list:

xml.providers.flowPersistenceProvider.file-provider.property.Flow Storage Directory

xml.providers.flowPersistenceProvider.file-provider.class

(this prevents NiFi Registry from adding the default filesystem provider to the providers.xml along with the git provider, which would be invalid)

CONCLUSION:

After making the changes above and restarting the NiFi Registry service, you should be able to enable a registry client in the NiFi Controller Settings, then start version control in the NiFi canvas and see your changes reflected in the remote git repository (such as GitHub). For more information about versioning and dev/ops workflows using the NiFi Flow Registry with git, check out Pierre Villard’s excellent article here.

[#2]

Note that 

providers.flowPersistenceProvider.file-provider.property.Flow Storage Directory

in step 7 needs to be

xml.providers.flowPersistenceProvider.file-provider.property.Flow Storage Directory

as shown in the picture beneath it.

[#3]

Thanks, @HorizonNet - I've updated the article with the correction!