Storing Apache NiFi Versioned Flows in a Git Repository

[#1]

Objective

With the release of NiFi Registry 0.2.0, flow contents can now be stored under a Git directory using the new GitFlowPersistenceProvider.

This tutorial walks you through how to configure this provider in NiFi Registry so that versioned flows in NiFi are automatically saved to a Git repository.

A video version of this tutorial can be seen here: https://youtu.be/kK7eVppg9Aw

Environment

This tutorial was tested using the following environment and components:

• Mac OS X 10.11.6
• Apache NiFi Registry 0.2.0
• Apache NiFi 1.7.1

GitFlowPersistenceProvider

Git Configuration

First, create a new GitHub repo:

then clone it locally using the git clone command (e.g. git clone https://github.com/andrewmlim/versioned_flows.git):

Next, go to GitHub’s “Developer settings” and create a new “Personal access token”:

NiFi Registry Configuration

In the ./conf/providers.xml file, configure the following properties:

• Set org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider as the qualified class name
• Set “Flow Storage Directory” to the directory where the repo was cloned
• Set "Remote to Push" to origin
• Set “Remote Access User” to your GitHub username
• Set “Remote Access Password” to the personal access token

Here is an example of these changes in providers.xml:

<flowPersistenceProvider>
 <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
 <property name="Flow Storage Directory">./versioned_flows</property>
 <property name="Remote To Push">origin</property>
 <property name="Remote Access User">andrewmlim</property>
 <property name="Remote Access Password">f1295e16f933d4468d948ea276372da8b0585bda</property>
</flowPersistenceProvider>

Note: The “Remote To Push” property specifies the name of the remote to automatically push to. This property is optional and if not specified, commits will remain in the local repository unless a push is performed manually.

Saving a Versioned Flow to the Git Repo

Start up NiFi Registry and create a bucket:

Start up a NiFi instance and connect to the Registry:

Create a process group. Start version control:

Save the flow:

In GitHub, you will see that the Bucket and Flow have been saved in your repo:

As shown, Buckets are represented as directories and Flow contents are stored as files in the Bucket directory they belong to. Flow snapshot histories are managed as Git commits, meaning only the latest version of Buckets and Flows exist in the Git directory.

Note: The commit message states "By NiFi Registry user: anonymous" since the environment was unsecured and there was no user logged into NiFi. The commit message would have the user's identity if secure.

Helpful Links

Here are some helpful links that were used as references for this article:

• Apache NiFi Registry Administration Guide - Git FlowPersistenceProvider
• bryanbende.com - Apache NiFi Registry 0.2.0

[#2]

Hi @Andrew Lim, great article! Thanks a lot. I have a question. Can I do the same with BitBucket? If yes, what will be the class for bitbucket? Thanks.

[#3]

@Marshal Tito, I think you can. I just did on GitLab. Those access tokens are used by all those hosted git providers as limited scope passwords. So it is a matter of username/password in the end.

BUT important:
> then clone it locally using the git clone command (e.g. git clone https://github.com/andrewmlim/versioned_flows.git😞

It is mandatory to clone the https url (not the git one as it will try to use your ssh keys in that case)

[#4]

Hi,

let me advertise an option for git clone using git+ssh. I have put together a Docker image which supports git cloning using git+ssh out-of-the box.

Give it a try and let me know how it works. Usage is as follows:

 docker run --name nifi-registry \
   -p 18080:18080 \
   -v ~/.ssh:/home/nifi/.ssh \
   -e 'FLOW_PROVIDER=git' \
   -e 'GIT_REMOTE_URL=git@github.com:michalklempa/docker-nifi-registry-example-flow.git' \
   -e 'GIT_CHECKOUT_BRANCH=example' \
   -e 'FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git' \
   -e 'FLOW_PROVIDER_GIT_REMOTE_TO_PUSH=origin' \
   -e 'GIT_CONFIG_USER_NAME=Michal Klempa' \
   -e 'GIT_CONFIG_USER_EMAIL=michal.klempa@gmail.com' \
   -d \
   michalklempa/nifi-registry:latest

or using ~/.ssh as a bind mount point:

 docker run --name nifi-registry \
   -p 18080:18080 \
   -e 'FLOW_PROVIDER=git' \
   -e 'GIT_REMOTE_URL=git@github.com:michalklempa/docker-nifi-registry-example-flow.git' \
   -e 'GIT_CHECKOUT_BRANCH=example' \
   -e 'FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git' \
   -e 'FLOW_PROVIDER_GIT_REMOTE_TO_PUSH=origin' \
   -e 'GIT_CONFIG_USER_NAME=Michal Klempa' \
   -e 'GIT_CONFIG_USER_EMAIL=michal.klempa@gmail.com' \
   -e 'SSH_PRIVATE_KEY='$(base64 -w 0 < ~/.ssh/id_rsa) \
   -e 'SSH_KNOWN_HOSTS='$(base64 -w 0 < ~/.ssh/known_hosts) \
   -e 'SSH_PRIVATE_KEY_PASSPHRASE=' \
   -d \
   michalklempa/nifi-registry:latest

Using HTTPS is also supported:

docker run --name nifi-registry \
   -p 18080:18080 \
   -e 'FLOW_PROVIDER=git' \
   -e 'GIT_REMOTE_URL=https://github.com/michalklempa/docker-nifi-registry-example-flow.git' \
   -e 'GIT_CHECKOUT_BRANCH=example' \
   -e 'FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git' \
   -e 'FLOW_PROVIDER_GIT_REMOTE_TO_PUSH=origin' \
   -e 'FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER=michalklempa' \
   -e 'FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD=thisisnotmypassword:)' \
   -e 'GIT_CONFIG_USER_NAME=Michal Klempa' \
   -e 'GIT_CONFIG_USER_EMAIL=michalklempa@gmail.com' \
   -d \
   michalklempa/nifi-registry:latest

Full documentation available at github:

https://github.com/michalklempa/docker-nifi-registry/#git-cloning-the-repository-at-startup

and docker image on dockerhub:

https://hub.docker.com/r/michalklempa/nifi-registry

[#5]

Hi, Let's pretend that I'm being forced against my will to run Nifi on Windows. The registry isn't supported on Windows and I can't run a VM. Can I track certain Nifi files in a regular old Git repository and expect to be able to switch between branches with a simple checkout? If so which files do I need to track? And how do I refresh the Nifi work space to reflect those changes?

[#6]

Hi @Andrew Lim thank you for this detailed and step-by-step information, exactly what I was looking for.

But I do have some questions which perhaps are a little weird caused by having no experience with github.

1. If I get this right the connection to github is "only" to store the content of each versioned flow outside the server where the registry is installed? So it is some kind of additional safty. Is this right?

2. Following your description one has to decide either using nifi-registry or github? Or did I get this wrong?

3. What if I try to connect to github and then want (only) to turn back to nifi-registry? Will this cause any predictable problems?

4. Is there any restriction in a clustered environment?

I would be greatful if you will answer this questions, thanks a lot!

[#7]

@alim  I am trying to configure git with the NIFI registry installation on windows(running bat script). I configured my GitFlowPersistanceProvider based on the details provided above and getting following error, can you please help me resolve this error.

Failed to start web server: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [C:\Users\ymathiva\Desktop\DnAPilotHub\Installs\nifi-registry-0.4.0-bin\nifi-registry-0.4.0\work\jetty\nifi-registry-web-api-0.4.0.war\webapp\WEB-INF\classes\org\apache\nifi\registry\web\api\AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizationService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/AuthorizationService.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getFlowPersistenceProvider' defined in class path resource [org/apache/nifi/registry/provider/StandardProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.flow.FlowPersistenceProvider]: Factory method 'getFlowPersistenceProvider' threw exception; nested exception is org.apache.nifi.registry.provider.ProviderFactoryException: Error creating FlowPersistenceProvider with class name: org.apache.nifi.registry.provider.flow.GitFlowPersistenceProvider
Shutting down...

[#8]

Hi @yuvapraveen_k 

NiFi Registry is not supported on Windows. You can see the system requirements here:

https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#system-requirements

[#9]

Hi @Andrew Lim, 

Thank you for your tutorials. I have been trying to restore the flow of Git for more than a week without success. Help me !!!! 🙂

[#10]

Hi,

setting up the NiFi Registry so that it automatically pushes to GitHub worked well following this guide.  Now that it is working, however, I'm bothered by this:

Note: The commit message states "By NiFi Registry user: anonymous" since the environment was unsecured and there was no user logged into NiFi. The commit message would have the user's identity if secure.

I have set up NiFi for OIDC authentication and everything works well on that end.  I still see "By NiFi Registry user: anonymous" in all commit messages.  Will that change if I enable https in the registry (using http at the moment)?  I'm asking because the message seems to talk about a "NiFi Registry user", but when using the NiFi UI, users do not authenticate against the registry as far as I can tell.

Thanks! Hans

[#11]

Hi, @alim 

NiFi fails to push to GitHub. Buckets appear in the folder specified in providers.xml but not synchronized with GitHub repository.

I use NiFi Registry 0.8.0 on Linux Mint Cinnamon 20

Logs are full of stuff like this:

2021-03-17 16:25:31,145 INFO [main] org.apache.nifi.registry.NiFiRegistry Registry initialization took 5822674415 nanoseconds (5 seconds).
2021-03-17 16:27:04,948 INFO [NiFi Registry Web Server-22] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Access tokens are only issued over HTTPS. Returning Conflict response.
2021-03-17 16:27:04,954 INFO [NiFi Registry Web Server-38] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: User authentication/authorization is only supported when running over HTTPS.. Returning Conflict response.

Everything is configured exactly as explained in the article.

P.S. Certificate authentication is not configured for access to Registery instance, could this be a problem in my case?