Automating Starting Services in Apache NiFi and Applying Parameters

[#1]

Automating Starting Services in Apache NiFi and Applying Parameters

Automate all the things! You can call these commands interactively or script all of them with awesome DevOps tools. @Andre Araujo and @dchaffey can tell you more about that.

Enable All NiFi Services on the Canvas

By running this three times, I get any stubborn ones or ones that needed something previously running. This could be put into a loop; check the status before trying again.

nifi pg-list
nifi pg-status
nifi pg-get-services

The NiFi CLI has interactive help available and also some good documentation:

NiFi CLI Toolkit Guide

/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services  -u http://edge2ai-1.dim.local:8080 --processGroupId root  

/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services  -u http://edge2ai-1.dim.local:8080 --processGroupId root  

/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services  -u http://edge2ai-1.dim.local:8080 --processGroupId root  

We could then start a process group if we wanted:

nifi pg-start -u http://edge2ai-1.dim.local:8080 -pgid 2c1860b3-7f21-36f4-a0b8-b415c652fc62

List all process groups

/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-list -u http://edge2ai-1.dim.local:8080

List Parameters

/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi list-param-contexts -u http://edge2ai-1.dim.local:8080 -verbose

Set parameters to set parameter context for a process group; you can loop to do all.

• pgid => parameter group id
• pcid => parameter context id

I need to put this in a shell or Python script:

/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-set-param-context -u http://edge2ai-1.dim.local:8080 -verbose -pgid 2c1860b3-7f21-36f4-a0b8-b415c652fc62  -pcid 39f0f296-0177-1000-ffff-ffffdccb6d90

Example

setupnifi.sh (Github Link)

You could also use the NiFi REST API or Dan's awesome Python API

NiPyApi: A Python Client SDK for Apache NiFi

References

• DevOps:  Working with Parameter Contexts

• NiFi Toolkit CLI
• No More Spaghetti Flows
• Report on this Apache NiFi
• Everything Apache Nifi

• Cloudera Data Platform - Using Apache NiFi REST API in the Public Cloud

• Using NiFi CLI to Restore NiFi Flows From Backups

• Automating the Building, Migration, Backup, Restore and Testing of Streaming Applications

• Apache NiFi Toolkit Guide

• An overview of Apache NiFi and Toolkit CLI deployments

• Automate workflow deployment in Apache NiFi with the NiFi Registry

• DevOps for Apache NiFi 1.7 and More