Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Rising Star

Overview

As you may or may not know, Nifi-Registry 0.2.0 (included in HDF 3.2) allows the storage of versioned flows in a git repository.

@Andrew Lim wrote a great article on how to configure this storage for nifi-registry as you can see here: https://community.hortonworks.com/articles/210286/storing-apache-nifi-versioned-flows-in-a-git-repos...

I thought I'd give a few pointers on how to do it in HDF 3.2, following these four simple steps:

  • Step 1: Setup your git repository
  • Step 2: Clone repository on HDF server
  • Step 3: Configure nifi-registry via Ambari
  • Step 4: Restart nifi-registry

Step 1: Setup your git repository

This step is essentially exactly what was highlighted by Andrew.

Create a new repository

92539-screen-shot-2018-09-24-at-24755-pm.png

Create a token

Go to to GitHub’s “Developer settings” and create a new “Personal access token”, as follows:

92540-screen-shot-2018-09-24-at-25250-pm.png

Step 2: Clone repository on HDF server

Connect to the server running nifi-registry on your hdf cluster. Make sure that git is installed; if it isn't run the following and follow instructions:

$ sudo yum install git

Go to the nifi_registry_internal_dir configured for your registry (/var/lib/nifi-registry by default) and clone your repo as the nifiregistry user:

$ sudo su nifiregistry
$ cd /var/lib/nifi-registry
$ mkdir git
$ cd git
$ git clone [address_of_your_repo]

Step 3: Configure nifi-registry via Ambari

The configuration for git is already in your providers.xml commented out. You will have to swap it for the default one that uses a local folder. In your nifi-registry configs tab, go to Advanced nifi-registry-providers-env and change it as follows:

92544-screen-shot-2018-09-24-at-30121-pm.png

The key changes should look like this:

  <!--
    <flowPersistenceProvider>
        <class>org.apache.nifi.registry.provider.flow.FileSystemFlowPersistenceProvider</class>
        <property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/flow_storage</property>
    </flowPersistenceProvider>
-->
   
    <flowPersistenceProvider>
        <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
        <property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/git/perso-recognition-nifi-flows</property>
        <property name="Remote To Push">origin</property>
        <property name="Remote Access User">paulvid</property>
        <property name="Remote Access Password">[your_key]</property>
    </flowPersistenceProvider>

Step 4: Restart nifi-registry

Using Ambari, restart nifi-registry:

92542-screen-shot-2018-09-24-at-30419-pm.png

Existing buckets will be automatically uploaded when you commit changes, and your repository will look like this:

92543-screen-shot-2018-09-24-at-30554-pm.png

7,738 Views
Comments
avatar
New Contributor

Great article @pvidal

Just a comment: It should be better to explicitly say that the [address_of_your_repo] in git clone SHOULD BE the https variant and not the ssh one. Otherwise it tries to use the ssh keys of the user that nifi-registry runs as and this is problematic.

avatar
Rising Star

Fair point @Aggelos Karalias!

avatar

@pvidal I am trying to configure git with the NIFI registry installation on windows(running bat script). I configured my GitFlowPersistanceProvider based on the details provided above and getting following error, can you please help me resolve this error.

 

 

Failed to start web server: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [C:\Users\ymathiva\Desktop\DnAPilotHub\Installs\nifi-registry-0.4.0-bin\nifi-registry-0.4.0\work\jetty\nifi-registry-web-api-0.4.0.war\webapp\WEB-INF\classes\org\apache\nifi\registry\web\api\AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizationService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/AuthorizationService.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getFlowPersistenceProvider' defined in class path resource [org/apache/nifi/registry/provider/StandardProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.flow.FlowPersistenceProvider]: Factory method 'getFlowPersistenceProvider' threw exception; nested exception is org.apache.nifi.registry.provider.ProviderFactoryException: Error creating FlowPersistenceProvider with class name: org.apache.nifi.registry.provider.flow.GitFlowPersistenceProvider
Shutting down...

avatar
Rising Star

@yuvapraveen_k What's your configuration exactly (if you could paste it without your github key, that'd be great)? Are you using Apache Nifi/Nifi-Registry or are you using HDF? Which versions are you running?

avatar
New Contributor

@pvidal @yuvapraveen_k I have the same issue as above

nifi 1.9.2, nifi registry 0.5, rhel 7.XX

 

providers.xml has not been changed apart from the steps above

flow storage directory = /opt/nifi-registry-0.5.0/gitdir

remote to push= origin

remote access user = user

remote access password = userpassword

 

other things in the conf that were there to begin with

flow storage directory = ./flow_storage

extension bundle storage directory=./extension_bundles

 

 

 

Version history
Last update:
‎08-17-2019 06:22 AM
Updated by:
Contributors