Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
avatar
Expert Contributor

OBJECTIVE:

Step-by step instructions for enabling the git persistence provider for NiFi in Cloudera Flow Management (CFM) on Cloudera Manager.

 

OVERVIEW:

The default install of early releases of Cloudera Flow Management (CFM) includes only basic configurations for NiFi Registry in Cloudera Manager.

This article explains how to reconfigure the Nifi Registry persistence providers to use git instead of the filesystem provider.

 

PREREQUISITES:

CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later

 

ADDING NiFi METRICS TO CM:

1. Pick a location where the Nifi Registry will store local git repository files. This location has to be accessible to the NiFi Registry service account (typically: nifiregistry). It must also be initialized as a git repository using git init and linked to a remote repository using git remote add origin. (Alternatively, an existing git repository can be cloned to create the folder using git clone). The substitution variable ${nifi.registry.working.directory} can be used to point to the install location for NiFi Registry (usually /var/lib/nifiregistry/ when NiFi registry is installed by Cloudera Manager).

2. From the NIFIREGISTRY Configuration tab in Cloudera Manager, search for “flowPersistenceProvider”

3. Locate the “NiFi Registry Advanced Configuration Snippet (Safety Valve) for staging/providers.xml” Safety Valve configuration.

4. Add the following Name/Value Pairs:

 

Name

 

Value (example)

 

Description

 

xml.providers.flowPersistenceProvider.git-provider.class

org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider

Provider java class for git - should always be org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider

xml.providers.flowPersistenceProvider.git-provider.property.Flow Storage Directory

${nifi.registry.working.directory}/git/flow_storage

Local directory for git project. Must be git initialized and have a remote defined. Use https link

xml.providers.flowPersistenceProvider.git-provider.property.Remote To Push

origin

Name of remote to push to - typically "origin"

xml.providers.flowPersistenceProvider.git-provider.property.Remote Access User

somegitusername

User account for git remote repository

xml.providers.flowPersistenceProvider.git-provider.property.Remote Access Password

1234mygitaccesstoken567

Personal access token for Github API

 

Sample XML

 

 

<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote Access Password</name>
<value>1234mygitaccesstoken567</value>
<description>Personal access token for Github API</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.class</name>
<value>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</value>
<description>Provider java class for git - should always be org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Flow Storage Directory</name>
<value>${nifi.registry.working.directory}/git/flow_storage</value>
<description>Local directory for git project. Must be git initialized and have a remote defined. Use https link</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote To Push</name>
<value>origin</value>
<description>Name of remote to push to - typically "origin"</description>
</property>
<property>
<name>xml.providers.flowPersistenceProvider.git-provider.property.Remote Access User</name>
<value>somegitusername</value>
<description>User account for git remote repository</description>
</property>

 

 

 

5. From the NIFIREGISTRY Configuration tab in Cloudera Manager, search for “providers ignored properties”

6. Locate the “providers ignored properties” (nifi.registry.providers.ignored) configuration.

7. Add the following two properties to the ignored list:

 

 

xml.providers.flowPersistenceProvider.file-provider.property.Flow Storage Directory
xml.providers.flowPersistenceProvider.file-provider.class

 

 

(this prevents NiFi Registry from adding the default filesystem provider to the providers.xml along with the git provider, which would be invalid)

 

NiFi Ignore Properties.png

 

CONCLUSION:

After making the changes above and restarting the NiFi Registry service, you should be able to enable a registry client in the NiFi Controller Settings, then start version control in the NiFi canvas and see your changes reflected in the remote git repository (such as GitHub). For more information about versioning and dev/ops workflows using the NiFi Flow Registry with git, check out Pierre Villard’s excellent article here.

 

NiFi Version Control.png

 

 

 

 

 

4,412 Views
Comments

Note that 

providers.flowPersistenceProvider.file-provider.property.Flow Storage Directory

in step 7 needs to be

xml.providers.flowPersistenceProvider.file-provider.property.Flow Storage Directory

as shown in the picture beneath it.

avatar
Expert Contributor

Thanks, @HorizonNet - I've updated the article with the correction!

Version history
Last update:
‎09-16-2022 01:45 AM
Updated by:
Contributors