Integrating Git to Nifi Registry using SSH Authentication

[#1]

Hi All,

We have installed Nifi and Nifi Registry 1.28 on an AWS ECS instance running on Fargate. The installation seems to be working fine, and both the applications can communicate with each other. We now wanted to set up Git Integration for Nifi Registry using ssh authentication. Can someone please help with detailed steps, article or content with this. 

We did try multiple options, but this is not successful.

Since our Nifi instance is on ECS Fargate, we tried cloning the repo using Docker Image and SSH key which was successful. However, when we pass the details in the providers.xml file, the instance doesn't come up.

Can someone please help with the exact steps and what properties and values to be passed in providers.xml file when using ssh authentication.

@steven-matison @MattWho : Your guidance and help will be much appreciated!!!

[#2]

@rj27, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @SAMSAL @Shelton who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@rj27 
To set up Git integration for Apache NiFi Registry using SSH authentication, you need to configure the NiFi Registry to use a Git-based flow persistence provider.

Analysis of Current Setup

• You have Apache NiFi 1.28 running on AWS ECS Fargate
• You have Apache NiFi Registry 1.28 running on AWS ECS Fargate
• Both applications are communicating with each other successfully
• You need to integrate NiFi Registry with Git using SSH authentication

Below are the detailed steps to achieve this on an AWS ECS instance running on Fargate with NiFi and NiFi Registry 1.28.

Detailed Steps for Git Integration

Step 1: Update NiFi Registry Configuration

1. Modify the nifi-registry.properties file in your container

Add the following properties to configure the Git flow persistence provider

Spoiler

# Git Configuration
nifi.registry.db.git.remote=true
nifi.registry.db.git.remote.to.push=true
nifi.registry.db.git.repository=/opt/nifi-registry/git-repository nifi.registry.db.git.flow.storage.directory=/opt/nifi-registry/flow-storage nifi.registry.db.git.remote.url=ssh://git@your-git-server:port/your-repo.git
nifi.registry.db.git.remote.branch=master

Step 2: Set Up SSH Keys for Authentication

1. Generate an SSH key pair inside your container

Spoiler

mkdir -p /opt/nifi-registry/.ssh
ssh-keygen -t rsa -b 4096 -C "nifi-registry@example.com" -f /opt/nifi-registry/.ssh/id_rsa -N ""

2. Add your public key to your Git repository's authorized keys (in GitHub, GitLab, etc.)

• Copy the contents of /opt/nifi-registry/.ssh/id_rsa.pub
• Add it to your Git provider as a deploy key or authentication key

3. Configure SSH client in the container

Spoiler

cat > /opt/nifi-registry/.ssh/config << EOF
Host your-git-server
     IdentityFile /opt/nifi-registry/.ssh/id_rsa
     StrictHostKeyChecking no
     UserKnownHostsFile /dev/null
EOF

4. Set proper permissions

Spoiler

chmod 700 /opt/nifi-registry/.ssh
chmod 600 /opt/nifi-registry/.ssh/id_rsa
chmod 644 /opt/nifi-registry/.ssh/id_rsa.pub
chmod 600 /opt/nifi-registry/.ssh/config

Step 3: Update ECS Task Definition for Persistence

1. Update your ECS task definition to include a volume for SSH keys and Git repository  validate the JSON's

Spoiler

"volumes": [
    {
   "name": "nifi-registry-git",
   "dockerVolumeConfiguration": {
      "scope": "task",
      "driver": "local",
      "labels": null,
      "autoprovision": true
     }
  }
]

2. Mount this volume in your container definition

Spoiler

"mountPoints": [
     {
        "sourceVolume": "nifi-registry-git",
        "containerPath": "/opt/nifi-registry/.ssh",
        "readOnly": false
      },
     { "sourceVolume": "nifi-registry-git",
        "containerPath": "/opt/nifi-registry/git-repository",
        "readOnly": false
     }
]

Step 4: Configure Git User Information

1. Set Git user configuration

Spoiler

git config --global user.name "NiFi Registry"
git config --global user.email "nifi-registry@example.com"

Step 5: Initialize the Git Repository

1. Initialize the local Git repository

Spoiler

mkdir -p /opt/nifi-registry/git-repository
cd /opt/nifi-registry/git-repository
git init
git remote add origin ssh://git@your-git-server:port/your-repository.git

2. Test the connection

Spoiler

ssh -T git@your-git-server

Step 6: Configure NiFi to Connect to NiFi Registry

1. In NiFi UI, configure the Registry Client:
   • Click on the hamburger menu (≡) in the top-right corner
   • Select "Controller Settings"
   • Go to the "Registry Clients" tab
   • Add a new Registry Client with:
     • Name: Git-Backed Registry
     • URL: http://your-nifi-registry:18080

Step 7: Restart NiFi Registry

1. Restart the NiFi Registry service to apply change

Spoiler

# If using systemd
systemctl restart nifi-registry

# If using the command line
./bin/nifi-registry.sh restart

# In AWS ECS, update the service to force new deployment
aws ecs update-service --cluster your-cluster --service your-nifi-registry-service --force-new-deployment

Troubleshooting

1. Check NiFi Registry logs for Git-related errors:

Spoiler

tail -f /opt/nifi-registry/logs/nifi-registry-app.log

2. Verify SSH connectivity

Spoiler

ssh -vT git@your-git-server

3. Common issues:

• Permission problems: Ensure the NiFi Registry user has appropriate permissions
• Known hosts: If StrictHostKeyChecking is on, you need to accept the host key first
• Firewall: Ensure outbound connections to the Git server are allowed from the ECS task
                                                  Important precautions 

• Security: Ensure the private key is stored securely and not exposed in the container image or logs.

• Automation: Consider using AWS Secrets Manager or Parameter Store to manage the SSH key and passphrase securely.

• Backup: Regularly back up your Git repository to avoid data loss.
  
  Happy hadooping

[#4]

Thank You @Shelton  for your revert. Will give this this a try and confirm back. Could you also please suggest the values to be passed in providers.xml file. Also, we are going to use EFS as our external volume and mount the same here. So do we add the same for keys instead of local. Lastly, I am using my personal user and email id for git configuration as of now. What is the best way suggested. I see that you have mentioned below user name and email for configuration. Is that a default user or we create a service user with same name. Please suggest

git config --global user.name "NiFi Registry"
git config --global user.email "nifi-registry@example.com"

[#5]

@rj27 
Some clarification on the git setup 
This is the author name that will appear in commit messages

Spoiler

Set the global Git username to "NiFi Registry" 

This is the email address associated with commits

Spoiler

Set the global Git email to "nifi-registry@example.com"  

Values to be passed 

Spoiler

<flowPersistenceProvider>
        <property name="Flow Storage Directory">./flow_storage</property>
       <property name="Git Remote To Push">origin</property>
       <property name="Git Remote Access User">username</property>
      <property name="Git Remote Access Password">password</property>
      <property name="Remote Clone Repository">https://git-repo-url/your-flow-repo.git</property> </flowPersistenceProvider>

[#6]

@Shelton - Also if you suggest the best way to set up CICD pipelines for 1.28 version considering we have GIT in place. How can I move my code and parameter context in higher environments. We have different set of nifi and nifi registry for each environmenet and we plan to maintain separate dev, qa and prod branches in same git repo. We want to set up GitHub Action workflows for the same. We have just adopted Nifi and are trying to figure out the best way to set up automated cicd pipelines for each environment considering all flows and parameter context are being taken care of.

[#7]

Thank you so much for helping me out.