question Integrating Git to Nifi Registry using SSH Authentication in Support Questions

Integrating Git to Nifi Registry using SSH Authentication

rj27 — Thu, 27 Feb 2025 10:01:18 GMT

Hi All,

We have installed Nifi and Nifi Registry 1.28 on an AWS ECS instance running on Fargate. The installation seems to be working fine, and both the applications can communicate with each other. We now wanted to set up Git Integration for Nifi Registry using ssh authentication. Can someone please help with detailed steps, article or content with this.

We did try multiple options, but this is not successful.

Since our Nifi instance is on ECS Fargate, we tried cloning the repo using Docker Image and SSH key which was successful. However, when we pass the details in the providers.xml file, the instance doesn't come up.

Can someone please help with the exact steps and what properties and values to be passed in providers.xml file when using ssh authentication.

@steven-matison @MattWho : Your guidance and help will be much appreciated!!!

Re: Integrating Git to Nifi Registry using SSH Authentication

VidyaSargur — Fri, 28 Feb 2025 06:41:31 GMT

@rj27, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @SAMSAL @Shelton who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Integrating Git to Nifi Registry using SSH Authentication

Shelton — Sat, 01 Mar 2025 19:46:39 GMT

@rj27
To set up Git integration for Apache NiFi Registry using SSH authentication, you need to configure the NiFi Registry to use a Git-based flow persistence provider.

Analysis of Current Setup

You have Apache NiFi 1.28 running on AWS ECS Fargate
You have Apache NiFi Registry 1.28 running on AWS ECS Fargate
Both applications are communicating with each other successfully
You need to integrate NiFi Registry with Git using SSH authentication

Below are the detailed steps to achieve this on an AWS ECS instance running on Fargate with NiFi and NiFi Registry 1.28.

Detailed Steps for Git Integration

Step 1: Update NiFi Registry Configuration

Modify the nifi-registry.properties file in your container

Add the following properties to configure the Git flow persistence provider

# Git Configuration
nifi.registry.db.git.remote=true
nifi.registry.db.git.remote.to.push=true
nifi.registry.db.git.repository=/opt/nifi-registry/git-repository nifi.registry.db.git.flow.storage.directory=/opt/nifi-registry/flow-storage nifi.registry.db.git.remote.url=ssh://git@your-git-server:port/your-repo.git
nifi.registry.db.git.remote.branch=master

Step 2: Set Up SSH Keys for Authentication

1. Generate an SSH key pair inside your container

mkdir -p /opt/nifi-registry/.ssh
ssh-keygen -t rsa -b 4096 -C "nifi-registry@example.com" -f /opt/nifi-registry/.ssh/id_rsa -N ""

2. Add your public key to your Git repository's authorized keys (in GitHub, GitLab, etc.)

Copy the contents of /opt/nifi-registry/.ssh/id_rsa.pub
Add it to your Git provider as a deploy key or authentication key

3. Configure SSH client in the container

cat > /opt/nifi-registry/.ssh/config << EOF
Host your-git-server
IdentityFile /opt/nifi-registry/.ssh/id_rsa
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
EOF

4. Set proper permissions

chmod 700 /opt/nifi-registry/.ssh
chmod 600 /opt/nifi-registry/.ssh/id_rsa
chmod 644 /opt/nifi-registry/.ssh/id_rsa.pub
chmod 600 /opt/nifi-registry/.ssh/config

Step 3: Update ECS Task Definition for Persistence

1. Update your ECS task definition to include a volume for SSH keys and Git repository validate the JSON's

"volumes": [
{
"name": "nifi-registry-git",
"dockerVolumeConfiguration": {
"scope": "task",
"driver": "local",
"labels": null,
"autoprovision": true
}
}
]

2. Mount this volume in your container definition

"mountPoints": [
{
"sourceVolume": "nifi-registry-git",
"containerPath": "/opt/nifi-registry/.ssh",
"readOnly": false
},
{ "sourceVolume": "nifi-registry-git",
"containerPath": "/opt/nifi-registry/git-repository",
"readOnly": false
}
]

Step 4: Configure Git User Information

Set Git user configuration

git config --global user.name "NiFi Registry"
git config --global user.email "nifi-registry@example.com"

Step 5: Initialize the Git Repository

Initialize the local Git repository

mkdir -p /opt/nifi-registry/git-repository
cd /opt/nifi-registry/git-repository
git init
git remote add origin ssh://git@your-git-server:port/your-repository.git

2. Test the connection

ssh -T git@your-git-server

Step 6: Configure NiFi to Connect to NiFi Registry

In NiFi UI, configure the Registry Client:
- Click on the hamburger menu (≡) in the top-right corner
- Select "Controller Settings"
- Go to the "Registry Clients" tab
- Add a new Registry Client with:
  - Name: Git-Backed Registry
  - URL: http://your-nifi-registry:18080

Step 7: Restart NiFi Registry

Restart the NiFi Registry service to apply change

# If using systemd
systemctl restart nifi-registry

# If using the command line
./bin/nifi-registry.sh restart

# In AWS ECS, update the service to force new deployment
aws ecs update-service --cluster your-cluster --service your-nifi-registry-service --force-new-deployment

Troubleshooting

1. Check NiFi Registry logs for Git-related errors:

tail -f /opt/nifi-registry/logs/nifi-registry-app.log

2. Verify SSH connectivity

ssh -vT git@your-git-server

3. Common issues:

Permission problems: Ensure the NiFi Registry user has appropriate permissions
Known hosts: If StrictHostKeyChecking is on, you need to accept the host key first
Firewall: Ensure outbound connections to the Git server are allowed from the ECS task
Important precautions

Security: Ensure the private key is stored securely and not exposed in the container image or logs.
Automation: Consider using AWS Secrets Manager or Parameter Store to manage the SSH key and passphrase securely.
Backup: Regularly back up your Git repository to avoid data loss.

Happy hadooping

Re: Integrating Git to Nifi Registry using SSH Authentication

rj27 — Sun, 02 Mar 2025 09:51:40 GMT

Thank You @Shelton for your revert. Will give this this a try and confirm back. Could you also please suggest the values to be passed in providers.xml file. Also, we are going to use EFS as our external volume and mount the same here. So do we add the same for keys instead of local. Lastly, I am using my personal user and email id for git configuration as of now. What is the best way suggested. I see that you have mentioned below user name and email for configuration. Is that a default user or we create a service user with same name. Please suggest

git config --global user.name "NiFi Registry"
git config --global user.email "nifi-registry@example.com"

Re: Integrating Git to Nifi Registry using SSH Authentication

rj27 — Sun, 02 Mar 2025 09:59:44 GMT

@Shelton - Also if you suggest the best way to set up CICD pipelines for 1.28 version considering we have GIT in place. How can I move my code and parameter context in higher environments. We have different set of nifi and nifi registry for each environmenet and we plan to maintain separate dev, qa and prod branches in same git repo. We want to set up GitHub Action workflows for the same. We have just adopted Nifi and are trying to figure out the best way to set up automated cicd pipelines for each environment considering all flows and parameter context are being taken care of.

Re: Integrating Git to Nifi Registry using SSH Authentication

Shelton — Sun, 02 Mar 2025 10:31:38 GMT

@rj27
Some clarification on the git setup
This is the author name that will appear in commit messages

Set the global Git username to "NiFi Registry"

This is the email address associated with commits

Set the global Git email to "nifi-registry@example.com"

Values to be passed

<flowPersistenceProvider>
<property name="Flow Storage Directory">./flow_storage</property>
<property name="Git Remote To Push">origin</property>
<property name="Git Remote Access User">username</property>
<property name="Git Remote Access Password">password</property>
<property name="Remote Clone Repository">https://git-repo-url/your-flow-repo.git</property> </flowPersistenceProvider>

Re: Integrating Git to Nifi Registry using SSH Authentication

RobertWilliams — Mon, 24 Mar 2025 10:58:52 GMT

Thank you so much for helping me out.