Created on 08-09-2018 04:10 PM - edited 08-17-2019 06:44 AM
With the release of NiFi Registry 0.2.0, flow contents can now be stored under a Git directory using the new
GitFlowPersistenceProvider
.
This tutorial walks you through how to configure this provider in NiFi Registry so that versioned flows in NiFi are automatically saved to a Git repository.
A video version of this tutorial can be seen here: https://youtu.be/kK7eVppg9Aw
This tutorial was tested using the following environment and components:
First, create a new GitHub repo:
then clone it locally using the
git clone
command (e.g. git clone https://github.com/andrewmlim/versioned_flows.git
):
Next, go to GitHub’s “Developer settings” and create a new “Personal access token”:
In the ./conf/providers.xml file, configure the following properties:
org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider
as the qualified class nameorigin
Here is an example of these changes in providers.xml:
<flowPersistenceProvider> <class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class> <property name="Flow Storage Directory">./versioned_flows</property> <property name="Remote To Push">origin</property> <property name="Remote Access User">andrewmlim</property> <property name="Remote Access Password">f1295e16f933d4468d948ea276372da8b0585bda</property> </flowPersistenceProvider>
Note: The “Remote To Push” property specifies the name of the remote to automatically push to. This property is optional and if not specified, commits will remain in the local repository unless a push is performed manually.
Start up NiFi Registry and create a bucket:
Start up a NiFi instance and connect to the Registry:
Create a process group. Start version control:
Save the flow:
In GitHub, you will see that the Bucket and Flow have been saved in your repo:
As shown, Buckets are represented as directories and Flow contents are stored as files in the Bucket directory they belong to. Flow snapshot histories are managed as Git commits, meaning only the latest version of Buckets and Flows exist in the Git directory.
Note: The commit message states "By NiFi Registry user: anonymous" since the environment was unsecured and there was no user logged into NiFi. The commit message would have the user's identity if secure.
Here are some helpful links that were used as references for this article:
Created on 02-13-2019 08:12 PM
Hi @Andrew Lim, great article! Thanks a lot. I have a question. Can I do the same with BitBucket? If yes, what will be the class for bitbucket? Thanks.
Created on 02-14-2019 02:45 PM
@Marshal Tito, I think you can. I just did on GitLab. Those access tokens are used by all those hosted git providers as limited scope passwords. So it is a matter of username/password in the end.
BUT important:
> then clone it locally using the git clone
command (e.g. git clone https://github.com/andrewmlim/versioned_flows.git
😞
It is mandatory to clone the https url (not the git one as it will try to use your ssh keys in that case)
Created on 04-01-2019 01:20 AM
Hi,
let me advertise an option for git clone using git+ssh. I have put together a Docker image which supports git cloning using git+ssh out-of-the box.
Give it a try and let me know how it works. Usage is as follows:
docker run --name nifi-registry \
-p 18080:18080 \
-v ~/.ssh:/home/nifi/.ssh \
-e 'FLOW_PROVIDER=git' \
-e 'GIT_REMOTE_URL=git@github.com:michalklempa/docker-nifi-registry-example-flow.git' \
-e 'GIT_CHECKOUT_BRANCH=example' \
-e 'FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git' \
-e 'FLOW_PROVIDER_GIT_REMOTE_TO_PUSH=origin' \
-e 'GIT_CONFIG_USER_NAME=Michal Klempa' \
-e 'GIT_CONFIG_USER_EMAIL=michal.klempa@gmail.com' \
-d \
michalklempa/nifi-registry:latest
or using ~/.ssh as a bind mount point:
docker run --name nifi-registry \
-p 18080:18080 \
-e 'FLOW_PROVIDER=git' \
-e 'GIT_REMOTE_URL=git@github.com:michalklempa/docker-nifi-registry-example-flow.git' \
-e 'GIT_CHECKOUT_BRANCH=example' \
-e 'FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git' \
-e 'FLOW_PROVIDER_GIT_REMOTE_TO_PUSH=origin' \
-e 'GIT_CONFIG_USER_NAME=Michal Klempa' \
-e 'GIT_CONFIG_USER_EMAIL=michal.klempa@gmail.com' \
-e 'SSH_PRIVATE_KEY='$(base64 -w 0 < ~/.ssh/id_rsa) \
-e 'SSH_KNOWN_HOSTS='$(base64 -w 0 < ~/.ssh/known_hosts) \
-e 'SSH_PRIVATE_KEY_PASSPHRASE=' \
-d \
michalklempa/nifi-registry:latest
Using HTTPS is also supported:
docker run --name nifi-registry \
-p 18080:18080 \
-e 'FLOW_PROVIDER=git' \
-e 'GIT_REMOTE_URL=https://github.com/michalklempa/docker-nifi-registry-example-flow.git' \
-e 'GIT_CHECKOUT_BRANCH=example' \
-e 'FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git' \
-e 'FLOW_PROVIDER_GIT_REMOTE_TO_PUSH=origin' \
-e 'FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER=michalklempa' \
-e 'FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD=thisisnotmypassword:)' \
-e 'GIT_CONFIG_USER_NAME=Michal Klempa' \
-e 'GIT_CONFIG_USER_EMAIL=michalklempa@gmail.com' \
-d \
michalklempa/nifi-registry:latest
Full documentation available at github:
https://github.com/michalklempa/docker-nifi-registry/#git-cloning-the-repository-at-startup
and docker image on dockerhub:
https://hub.docker.com/r/michalklempa/nifi-registry
Created on 04-25-2019 12:50 PM
Hi, Let's pretend that I'm being forced against my will to run Nifi on Windows. The registry isn't supported on Windows and I can't run a VM. Can I track certain Nifi files in a regular old Git repository and expect to be able to switch between branches with a simple checkout? If so which files do I need to track? And how do I refresh the Nifi work space to reflect those changes?
Created on 07-24-2019 06:59 AM
Hi @Andrew Lim thank you for this detailed and step-by-step information, exactly what I was looking for.
But I do have some questions which perhaps are a little weird caused by having no experience with github.
1. If I get this right the connection to github is "only" to store the content of each versioned flow outside the server where the registry is installed? So it is some kind of additional safty. Is this right?
2. Following your description one has to decide either using nifi-registry or github? Or did I get this wrong?
3. What if I try to connect to github and then want (only) to turn back to nifi-registry? Will this cause any predictable problems?
4. Is there any restriction in a clustered environment?
I would be greatful if you will answer this questions, thanks a lot!
Created on 09-04-2019 07:17 AM
@alim I am trying to configure git with the NIFI registry installation on windows(running bat script). I configured my GitFlowPersistanceProvider based on the details provided above and getting following error, can you please help me resolve this error.
Failed to start web server: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [C:\Users\ymathiva\Desktop\DnAPilotHub\Installs\nifi-registry-0.4.0-bin\nifi-registry-0.4.0\work\jetty\nifi-registry-web-api-0.4.0.war\webapp\WEB-INF\classes\org\apache\nifi\registry\web\api\AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizationService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/AuthorizationService.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/C:/Users/ymathiva/Desktop/DnAPilotHub/Installs/nifi-registry-0.4.0-bin/nifi-registry-0.4.0/work/jetty/nifi-registry-web-api-0.4.0.war/webapp/WEB-INF/lib/nifi-registry-framework-0.4.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getFlowPersistenceProvider' defined in class path resource [org/apache/nifi/registry/provider/StandardProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.flow.FlowPersistenceProvider]: Factory method 'getFlowPersistenceProvider' threw exception; nested exception is org.apache.nifi.registry.provider.ProviderFactoryException: Error creating FlowPersistenceProvider with class name: org.apache.nifi.registry.provider.flow.GitFlowPersistenceProvider
Shutting down...
Created on 05-04-2020 11:46 AM
NiFi Registry is not supported on Windows. You can see the system requirements here:
https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#system-requirements
Created on 05-12-2020 01:15 PM
Hi @Andrew Lim,
Thank you for your tutorials. I have been trying to restore the flow of Git for more than a week without success. Help me !!!! 🙂
Created on 09-07-2020 10:02 PM
Hi,
setting up the NiFi Registry so that it automatically pushes to GitHub worked well following this guide. Now that it is working, however, I'm bothered by this:
Note: The commit message states "By NiFi Registry user: anonymous" since the environment was unsecured and there was no user logged into NiFi. The commit message would have the user's identity if secure.
I have set up NiFi for OIDC authentication and everything works well on that end. I still see "By NiFi Registry user: anonymous" in all commit messages. Will that change if I enable https in the registry (using http at the moment)? I'm asking because the message seems to talk about a "NiFi Registry user", but when using the NiFi UI, users do not authenticate against the registry as far as I can tell.
Thanks! Hans
Created on 03-17-2021 07:50 AM
Hi, @alim
NiFi fails to push to GitHub. Buckets appear in the folder specified in providers.xml but not synchronized with GitHub repository.
I use NiFi Registry 0.8.0 on Linux Mint Cinnamon 20
Logs are full of stuff like this:
2021-03-17 16:25:31,145 INFO [main] org.apache.nifi.registry.NiFiRegistry Registry initialization took 5822674415 nanoseconds (5 seconds).
2021-03-17 16:27:04,948 INFO [NiFi Registry Web Server-22] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Access tokens are only issued over HTTPS. Returning Conflict response.
2021-03-17 16:27:04,954 INFO [NiFi Registry Web Server-38] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: User authentication/authorization is only supported when running over HTTPS.. Returning Conflict response.
Everything is configured exactly as explained in the article.
P.S. Certificate authentication is not configured for access to Registery instance, could this be a problem in my case?