Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Cloudera Employee

Overview

As you may or may not know, Nifi-Registry 0.2.0 (included in HDF 3.2) allows the storage of versioned flows in a git repository.

@Andrew Lim wrote a great article on how to configure this storage for nifi-registry as you can see here: https://community.hortonworks.com/articles/210286/storing-apache-nifi-versioned-flows-in-a-git-repos...

I thought I'd give a few pointers on how to do it in HDF 3.2, following these four simple steps:

  • Step 1: Setup your git repository
  • Step 2: Clone repository on HDF server
  • Step 3: Configure nifi-registry via Ambari
  • Step 4: Restart nifi-registry

Step 1: Setup your git repository

This step is essentially exactly what was highlighted by Andrew.

Create a new repository

92539-screen-shot-2018-09-24-at-24755-pm.png

Create a token

Go to to GitHub’s “Developer settings” and create a new “Personal access token”, as follows:

92540-screen-shot-2018-09-24-at-25250-pm.png

Step 2: Clone repository on HDF server

Connect to the server running nifi-registry on your hdf cluster. Make sure that git is installed; if it isn't run the following and follow instructions:

$ sudo yum install git

Go to the nifi_registry_internal_dir configured for your registry (/var/lib/nifi-registry by default) and clone your repo as the nifiregistry user:

$ sudo su nifiregistry
$ cd /var/lib/nifi-registry
$ mkdir git
$ cd git
$ git clone [address_of_your_repo]

Step 3: Configure nifi-registry via Ambari

The configuration for git is already in your providers.xml commented out. You will have to swap it for the default one that uses a local folder. In your nifi-registry configs tab, go to Advanced nifi-registry-providers-env and change it as follows:

92544-screen-shot-2018-09-24-at-30121-pm.png

The key changes should look like this:

  <!--
    <flowPersistenceProvider>
        <class>org.apache.nifi.registry.provider.flow.FileSystemFlowPersistenceProvider</class>
        <property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/flow_storage</property>
    </flowPersistenceProvider>
-->
   
    <flowPersistenceProvider>
        <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
        <property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/git/perso-recognition-nifi-flows</property>
        <property name="Remote To Push">origin</property>
        <property name="Remote Access User">paulvid</property>
        <property name="Remote Access Password">[your_key]</property>
    </flowPersistenceProvider>

Step 4: Restart nifi-registry

Using Ambari, restart nifi-registry:

92542-screen-shot-2018-09-24-at-30419-pm.png

Existing buckets will be automatically uploaded when you commit changes, and your repository will look like this:

92543-screen-shot-2018-09-24-at-30554-pm.png

2,417 Views
Comments
Not applicable

Great article @pvidal

Just a comment: It should be better to explicitly say that the [address_of_your_repo] in git clone SHOULD BE the https variant and not the ssh one. Otherwise it tries to use the ssh keys of the user that nifi-registry runs as and this is problematic.

Cloudera Employee

Fair point @Aggelos Karalias!

New Contributor

@pvidal I am trying to configure git with the NIFI registry installation on windows(running bat script). I configured my GitFlowPersistanceProvider based on the details provided above and getting following error, can you please help me resolve this error.

 

 

Failed to start web server: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [C:\Users\ymathiva\Desktop\DnAPilotHub\Installs\nifi-registry-0.4.0-bin\nifi-registry-0.4.0\work\jetty\nifi-registry-web-api-0.4.0.war\webapp\WEB-INF\classes\org\apache\nifi\registry\web\api\AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizationService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/AuthorizationService.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getFlowPersistenceProvider' defined in class path resource [org/apache/nifi/registry/provider/StandardProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.flow.FlowPersistenceProvider]: Factory method 'getFlowPersistenceProvider' threw exception; nested exception is org.apache.nifi.registry.provider.ProviderFactoryException: Error creating FlowPersistenceProvider with class name: org.apache.nifi.registry.provider.flow.GitFlowPersistenceProvider
Shutting down...

Cloudera Employee

@yuvapraveen_k What's your configuration exactly (if you could paste it without your github key, that'd be great)? Are you using Apache Nifi/Nifi-Registry or are you using HDF? Which versions are you running?

New Contributor

@pvidal @yuvapraveen_k I have the same issue as above

nifi 1.9.2, nifi registry 0.5, rhel 7.XX

 

providers.xml has not been changed apart from the steps above

flow storage directory = /opt/nifi-registry-0.5.0/gitdir

remote to push= origin

remote access user = user

remote access password = userpassword

 

other things in the conf that were there to begin with

flow storage directory = ./flow_storage

extension bundle storage directory=./extension_bundles

 

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 06:22 AM
Updated by:
 
Contributors
Top Kudoed Authors