NiFi SDLC Automation in Python with NiPyApi - Part 1

[#1]

In this article I will explain an easy way to automate some basic tasks in NiFi as an introduction to NiPyApi, an automation package for Apache NiFi and its sub-projects.

Environment Setup Requirements

• You will need a Python environment
  • 2.7 or 3.6 are tested, and most computers come with one of these
  • or you can create a virtualenv
  • or you can install Python on OSX using homebrew
• You will need a recent internet browser, given you're reading this I assume you have one - I'm using Chrome
• You will need NiFi services to test against - if you have Docker installed you can use the following commands to prepare NiFi & NiFi Registry services:

user$: curl https://raw.githubusercontent.com/Chaffelson/nipyapi/master/test_env_config/docker_compose_latest/do... | docker-compose -f - up -d
user$: docker ps

Docker will download and start the NiFi containers, and show you the details:

You should be able to browse to both NiFi and NiFi-Registry at the following URLs:

• http://localhost:8080/nifi/
• http://localhost:18080/nifi-registry/

Installing NiPyApi

Installing NiPyApi is very easy, and done by the usual Python package distribution manager called Pip:

user$: pip install nipyapi

NiPyApi will install along with it's package dependencies, much like a linux package - don't worry about the dependencies, it'll look like this when it's done:

Once that completes, go ahead and start an interactive python session on your command line and run a test command:

user$: python
>>> from nipyapi import config, canvas
>>> config.nifi_config.host
'http://localhost:8080/nifi-api'
>>> canvas.get_root_pg_id()
'4e8d8f99-0161-1000-fa6f-724e5873aebc'

NiPyApi will look for a NiFi environment on the usual port, or you can change this in nipyapi.config shown above.

Congratulations! You have just commanded the NiFi API in less than 5 lines of code.

Investigating the Package

Now we can try using a few of the NiPyApi commands to interact with the NiFi environment - while the entire NiFi and NiFi-Registry APIs are implemented, only some of the calls are surfaced for common use - you can find out about them in great detail either through the online documentation at ReadTheDocs, or by investigating the Github Repo.
For now, try looking at the console documentation of the nipyapi.canvas functions using the help() command:

>>> help(canvas)
Help on module nipyapi.canvas in nipyapi:
NAME
    nipyapi.canvas
FILE
    /Users/dchaffey/.virtualenvs/tmp-167d86bd91b19b09/lib/python2.7/site-packages/nipyapi/canvas.py
DESCRIPTION
    For interactions with the NiFi Canvas
    STATUS: Work in Progress to determine pythonic datamodel
FUNCTIONS
    create_process_group(parent_pg, new_pg_name, location)
        Creates a new PG with a given name under the provided parent PG
        :param parent_pg: ProcessGroupEntity object of the parent PG
        :param new_pg_name: String to name the new PG
        :param location: Tuple of (x,y) coordinates to place the new PG
        :return: ProcessGroupEntity of the new PG
...

You can see there are a lot of functions here that you can use to complete tasks against NiFi, and there are even more in the nipyapi.templates and nipyapi.versioning modules.

Trying an Automation Script

There is a handy interactive Demo built into NiPyApi, and this time we're also going to use the new NiFi-Registry as well.

It will procedurally generate a Process Group containing a Processor in NiFi, and then put them under Version Control in the NiFi Registry. It will then also clone the version from one Registry Bucket to another, simulating code promotion:

Note that if you did not use the supplied Docker configuration above, you may have to modify the script to connect to your NiFi and NiFi-Registry environments.

>>> from nipyapi.demo.console import *
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__', 'bucket_0', 'bucket_1', 'canvas', 'config', 'process_group_0', 'processor_0', 'reg_client_0', 'ver_flow_0', 'ver_flow_1', 'ver_flow_info_0', 'ver_flow_snapshot_0', 'ver_flow_snapshot_1']

You can see here a number of NiFi and Registry objects have been created for you by the automation script as described.

You can take a look at the script and how it's using the NiPyApi functions on Github.

If you head over to your NiFi and NiFi-Registry GUI, you can explore the objects and try the new features out for yourself.

Happy Coding!