Nifi toolkit command for GitLabFlowRegistry

shiva239 — Wed, 06 Nov 2024 22:23:32 GMT

I was able to setup GitLabFlowRegistry in controller settings successfully (which is new feature in Nifi 2.0)

From Nifi UI, I am able to import Nifi flow directly from Gitlab repository. After modifications, I was able to successfully commit into Gitlab repo, from Nifi UI.

Do we have any toolkit commands to perform these operations? Currently we are using toolkit commands to deploy Nifi flow changes via CICD pipeline. Can I achieve the same with new GitLabFlowRegistry?

Appreciate any help/pointers.

Thanks

Re: Nifi toolkit command for GitLabFlowRegistry

Shelton — Sun, 23 Mar 2025 12:23:13 GMT

@shiva239

You've set up GitLabFlowRegistry in NiFi 2.0's controller settings and can import flows from GitLab and commit changes back through the UI your existing CI/CD pipeline uses NiFi toolkit commands.
As of NiFi 2.0, there aren't direct toolkit commands specifically designed for the new GitLabFlowRegistry feature someone should correct me here. However, you have a few options to achieve what you need.

Option 1: Use the NiFi REST API
The GitLabFlowRegistry operations that you perform through the UI are backed by REST API endpoints. You can integrate these REST API calls into your CI/CD pipeline

# Example REST API call to import/sync from Git repository
curl -X POST "https://your-nifi-host:port/nifi-api/versions/process-groups/{processGroupId}/flow-registry-sync" \
-H "Content-Type: application/json" \
-d '{"registryId":"your-gitlab-registry-id","bucketId":"your-bucket-id","flowId":"your-flow-id","version":1}' \
--key your-key.key --cert your-cert.crt --cacert your-ca.crt

Option 2: Hybrid Approach
Use a combination of existing toolkit commands and REST API calls:

Use toolkit commands for basic operations (deployment to a target NiFi instance)
Use REST API calls for GitLabFlowRegistry-specific operations

Option 3: Create Custom Scripts
You could create wrapper scripts that combine the toolkit functionality with the necessary REST API calls for GitLabFlowRegistry operations:

#!/bin/bash
# Custom script to sync from GitLabFlowRegistry and deploy

# First pull from Git registry via REST API
curl -X POST "https://your-nifi-host:port/nifi-api/versions/process-groups/{processGroupId}/flow-registry-sync" \
-H "Content-Type: application/json" \
-d '{"registryId":"your-gitlab-registry-id","bucketId":"your-bucket-id","flowId":"your-flow-id","version":1}' \
--key your-key.key --cert your-cert.crt --cacert your-ca.crt

# Then use toolkit commands for additional operations as needed
./bin/cli.sh nifi pg-import ...

I recommend adopting Option 3 (custom scripts) for your CI/CD pipeline. This approach:

Takes advantage of the new GitLabFlowRegistry feature
Maintains compatibility with your existing toolkit-based CI/CD pipeline
Provides flexibility to customize the process based on your specific requirements

For implementing the REST API calls, consult the NiFi API documentation for the full set of endpoints related to the new GitLabFlowRegistry functionality. The NiFi Admin Guide for version 2.0 should also have details on these new REST endpoints.

Happy hadooping

Re: Nifi toolkit command for GitLabFlowRegistry

shiva239 — Thu, 27 Mar 2025 22:22:52 GMT

Thanks @Shelton for details!! I will try Option 3 in our pipeline shell script. Will let you know if any further issues.

question Re: Nifi toolkit command for GitLabFlowRegistry in Support Questions

Nifi toolkit command for GitLabFlowRegistry

Re: Nifi toolkit command for GitLabFlowRegistry

Re: Nifi toolkit command for GitLabFlowRegistry