Nifi toolkit command for GitLabFlowRegistry

[#1]

I was able to setup GitLabFlowRegistry in controller settings successfully (which is new feature in Nifi 2.0)

From Nifi UI, I am able to import Nifi flow directly from Gitlab repository.  After modifications, I was able to successfully commit into Gitlab repo, from Nifi UI.

Do we have any toolkit commands to perform these operations?  Currently we are using toolkit commands to deploy Nifi flow changes via CICD pipeline.  Can I achieve the same with new GitLabFlowRegistry?  

Appreciate any help/pointers.

Thanks

[#2 · Accepted solution]

@shiva239 

You've set up GitLabFlowRegistry in NiFi 2.0's controller settings and can import flows from GitLab and commit changes back through the UI your existing CI/CD pipeline uses NiFi toolkit commands.
As of NiFi 2.0, there aren't direct toolkit commands specifically designed for the new GitLabFlowRegistry feature someone should correct me here. However, you have a few options to achieve what you need.

Option 1: Use the NiFi REST API
The GitLabFlowRegistry operations that you perform through the UI are backed by REST API endpoints. You can integrate these REST API calls into your CI/CD pipeline

Spoiler

# Example REST API call to import/sync from Git repository
curl -X POST "https://your-nifi-host:port/nifi-api/versions/process-groups/{processGroupId}/flow-registry-sync" \
-H "Content-Type: application/json" \
-d '{"registryId":"your-gitlab-registry-id","bucketId":"your-bucket-id","flowId":"your-flow-id","version":1}' \
--key your-key.key --cert your-cert.crt --cacert your-ca.crt

Option 2: Hybrid Approach
Use a combination of existing toolkit commands and REST API calls:

1. Use toolkit commands for basic operations (deployment to a target NiFi instance)
2. Use REST API calls for GitLabFlowRegistry-specific operations

Option 3: Create Custom Scripts
You could create wrapper scripts that combine the toolkit functionality with the necessary REST API calls for GitLabFlowRegistry operations:

Spoiler

#!/bin/bash
# Custom script to sync from GitLabFlowRegistry and deploy

# First pull from Git registry via REST API
curl -X POST "https://your-nifi-host:port/nifi-api/versions/process-groups/{processGroupId}/flow-registry-sync" \
-H "Content-Type: application/json" \
-d '{"registryId":"your-gitlab-registry-id","bucketId":"your-bucket-id","flowId":"your-flow-id","version":1}' \
--key your-key.key --cert your-cert.crt --cacert your-ca.crt

# Then use toolkit commands for additional operations as needed
./bin/cli.sh nifi pg-import ...

I recommend adopting Option 3 (custom scripts) for your CI/CD pipeline. This approach:

1. Takes advantage of the new GitLabFlowRegistry feature
2. Maintains compatibility with your existing toolkit-based CI/CD pipeline
3. Provides flexibility to customize the process based on your specific requirements

For implementing the REST API calls, consult the NiFi API documentation for the full set of endpoints related to the new GitLabFlowRegistry functionality. The NiFi Admin Guide for version 2.0 should also have details on these new REST endpoints.

Happy hadooping 

[#3]

Thanks @Shelton for details!!  I will try Option 3 in our pipeline shell script.   Will let you know if any further issues.