How to configure git for Nifi Registry in HDF 3.2

[#1]

Overview

As you may or may not know, Nifi-Registry 0.2.0 (included in HDF 3.2) allows the storage of versioned flows in a git repository.

@Andrew Lim wrote a great article on how to configure this storage for nifi-registry as you can see here: https://community.hortonworks.com/articles/210286/storing-apache-nifi-versioned-flows-in-a-git-repos...

I thought I'd give a few pointers on how to do it in HDF 3.2, following these four simple steps:

• Step 1: Setup your git repository
• Step 2: Clone repository on HDF server
• Step 3: Configure nifi-registry via Ambari
• Step 4: Restart nifi-registry

Step 1: Setup your git repository

This step is essentially exactly what was highlighted by Andrew.

Create a new repository

Create a token

Go to to GitHub’s “Developer settings” and create a new “Personal access token”, as follows:

Step 2: Clone repository on HDF server

Connect to the server running nifi-registry on your hdf cluster. Make sure that git is installed; if it isn't run the following and follow instructions:

$ sudo yum install git

Go to the nifi_registry_internal_dir configured for your registry (/var/lib/nifi-registry by default) and clone your repo as the nifiregistry user:

$ sudo su nifiregistry
$ cd /var/lib/nifi-registry
$ mkdir git
$ cd git
$ git clone [address_of_your_repo]

Step 3: Configure nifi-registry via Ambari

The configuration for git is already in your providers.xml commented out. You will have to swap it for the default one that uses a local folder. In your nifi-registry configs tab, go to Advanced nifi-registry-providers-env and change it as follows:

The key changes should look like this:

  <!--
    <flowPersistenceProvider>
        <class>org.apache.nifi.registry.provider.flow.FileSystemFlowPersistenceProvider</class>
        <property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/flow_storage</property>
    </flowPersistenceProvider>
-->
   
    <flowPersistenceProvider>
        <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
        <property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/git/perso-recognition-nifi-flows</property>
        <property name="Remote To Push">origin</property>
        <property name="Remote Access User">paulvid</property>
        <property name="Remote Access Password">[your_key]</property>
    </flowPersistenceProvider>

Step 4: Restart nifi-registry

Using Ambari, restart nifi-registry:

Existing buckets will be automatically uploaded when you commit changes, and your repository will look like this:

[#2]

Great article @pvidal

Just a comment: It should be better to explicitly say that the [address_of_your_repo] in git clone SHOULD BE the https variant and not the ssh one. Otherwise it tries to use the ssh keys of the user that nifi-registry runs as and this is problematic.

[#3]

Fair point @Aggelos Karalias!

[#4]

@pvidal I am trying to configure git with the NIFI registry installation on windows(running bat script). I configured my GitFlowPersistanceProvider based on the details provided above and getting following error, can you please help me resolve this error.

Failed to start web server: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [C:\Users\ymathiva\Desktop\DnAPilotHub\Installs\nifi-registry-0.4.0-bin\nifi-registry-0.4.0\work\jetty\nifi-registry-web-api-0.4.0.war\webapp\WEB-INF\classes\org\apache\nifi\registry\web\api\AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizationService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/AuthorizationService.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getFlowPersistenceProvider' defined in class path resource [org/apache/nifi/registry/provider/StandardProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.flow.FlowPersistenceProvider]: Factory method 'getFlowPersistenceProvider' threw exception; nested exception is org.apache.nifi.registry.provider.ProviderFactoryException: Error creating FlowPersistenceProvider with class name: org.apache.nifi.registry.provider.flow.GitFlowPersistenceProvider
Shutting down...

[#5]

@yuvapraveen_k What's your configuration exactly (if you could paste it without your github key, that'd be great)? Are you using Apache Nifi/Nifi-Registry or are you using HDF? Which versions are you running?

[#6]

@pvidal @yuvapraveen_k I have the same issue as above

nifi 1.9.2, nifi registry 0.5, rhel 7.XX

providers.xml has not been changed apart from the steps above

flow storage directory = /opt/nifi-registry-0.5.0/gitdir

remote to push= origin

remote access user = user

remote access password = userpassword

other things in the conf that were there to begin with

flow storage directory = ./flow_storage

extension bundle storage directory=./extension_bundles