Creating NiFi Template via Rest API

[#1]

The objective of this article is to clearly demonstrate how to use NiFi's REST API to create a template on a secured cluster. I plan to publish multiple articles on using NiFi rest api based on community feedback around the use cases in the field. No pie in the sky type of knowledge share. Adding complexity to the workflows as I publish new articles and how they look via rest api publishing.

Use Case Setup:

Create template of my process group UVindex via rest api. UVindex resides within a process group hierarchy

Root->SunileManjeeProcessGroup->RestAPI Test->UNIndex

UNindex is a simply process groups which polls Chiago uv index data every 5 minutes and stores on hdfs. CurrentWeatherData is in the same process group but for this use case I only want to create template for UVIndex process group.

Steps

1. Fetch Access Token
2. Fetch Client ID
3. Fetch Process Group attributes
4. Create Snippet
5. Create Template

Step 1 - Fetch Access Token

Access token is required for rest api interactions on secured nifi cluster. Token will be used in the header for all rest api interactions

-H "Authorization: Bearer myLongToken

API

/nifi-api/access/token

Fetch Access Token

curl -X POST -H "Accept: */*" -H "Accept-Encoding: gzip" -H "Accept-Language: en-US,en;q=0.5" -H "Connection: keep-alive" -H "Content-Length: 38" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -H "Host: https://nifi.com:9091" -H "Referer: https://nifi.com:9091/nifi/login" -H "X-Requested-With: XMLHttpRequest" -d 'username=myUserName&password=YourPasswordHere' https://nifi.com:9091/nifi-api/access/token | gunzip

Step 2

Fetch your client ID . Client ID will be used for all rest api interactions

curl -X GET -H "Accept: */*" -H "Accept-Encoding: gzip" -H "Accept-Language: en-US,en;q=0.5" -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1" -H "Connection: keep-alive" -H "Host: hdf20-0.field.hortonworks.com:9091" -H "Referer: https://nifi.com:9091/nifi/login" -H "X-Requested-With: XMLHttpRequest" https://nifi.com:9091/nifi-api/flow/client-id | gunzip

Step 3- Fetch Process Group Attributes

Fetch the UUID of the process group. To create a template a UUID of the object you want to create a template for is required.

API

/flow/process-groups/parentProcessGroupUUID

Or Rest API..Here my parent process group UUID is

8c3d29aa-d1ea-1c85-ffff-ffffe6b3853a

Fetch Process Group Info

curl -X GET -H "Accept: */*" -H "Accept-Encoding: gzip" -H "Accept-Language: en-US,en;q=0.5" -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1OaUZpIFVzZXI" -H "Connection: keep-alive" -H "Host: https://nifi.com:9091" -H "Referer: https://nifi.com:9091/nifi/login" -H "X-Requested-With: XMLHttpRequest" https://nifi.com:9091/nifi-api/flow/process-groups/8c3d29aa-d1ea-1c85-ffff-ffffe6b3853a | gunzip

Which returns large json object. Here you can parse the json object for the name of the process group (UVindex) and grab the UUID and version.

Step 4 - Create Snippet

To create a template, first a snippet must be created . The snippet ID will be used to name and create template. All details of what a template should be composed of are within a snippet

API

/nifi-api/snippets

For the payload populate PartentGroupID using the parent process group UUID. Under "processGroups" populate with UUID of the process group you want to create a template of, your client ID, and version captured in step 3. You may have connections or other objects which may require values within the payload. For this template it does not. In follow up articles I will create higher complexity work flows which will require such objection to be populated during snippet creation.

{
  "snippet": {
    "parentGroupId": "8c3d29aa-d1ea-1c85-ffff-ffffe6b3853a",
    "processors": {},
    "funnels": {},
    "inputPorts": {},
    "outputPorts": {},
    "remoteProcessGroups": {},
    "processGroups": {
      "d8b73f15-e918-1afc-9db2-e7cdfbf19076": {
        "clientId": "980e3bca-b76a-183f-2e3b-5e75dccd3aec",
        "version": 1
      }
    },
    "connections": {},
    "labels": {}
  }
}

Example

curl -X POST -H "Accept: application/json, text/javascript, */*; q=0.01" -H "Accept-Encoding: gzip" -H "Accept-Language: en-US,en;q=0.5" -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzd" -H "Connection: keep-alive" -H "Content-Length: 305" -H "Content-Type: application/json" -H "Host: nifi.com:9091" -H "Referer: https://hdf20-0.field.hortonworks.com:9091/nifi/login" -H "X-Requested-With: XMLHttpRequest" -d '{"snippet":{"parentGroupId":"8c3d29aa-d1ea-1c85-ffff-ffffe6b3853a","processors":{},"funnels":{},"inputPorts":{},"outputPorts":{},"remoteProcessGroups":{},"processGroups":{"d8b73f15-e918-1afc-9db2-e7cdfbf19076":{"clientId":"980e3bca-b76a-183f-2e3b-5e75dccd3aec","version":1}},"connections":{},"labels":{}}}' https://nifi.com:9091/nifi-api/snippets | gunzip

Output

{
  "snippet": {
    "id": "980e46c5-b76a-183f-ffff-ffffb4770059",
    "uri": "https://nifi.com:9091/nifi-api/process-groups/8c3d29aa-d1ea-1c85-ffff-ffffe6b3853a/snippets/980e46c5-b76a-183f-ffff-ffffb4770059",
    "parentGroupId": "8c3d29aa-d1ea-1c85-ffff-ffffe6b3853a",
    "processGroups": {
      "d8b73f15-e918-1afc-9db2-e7cdfbf19076": {
        "clientId": "980e3bca-b76a-183f-2e3b-5e75dccd3aec",
        "version": 1
      }
    },
    "remoteProcessGroups": {},
    "processors": {},
    "inputPorts": {},
    "outputPorts": {},
    "connections": {},
    "labels": {},
    "funnels": {}
  }
}

Fetch the snippet ID from the output

Step 5 - Create Template

Using the snippet ID from step 4 create a template named "My Template"

Payload

{
  "name": "My Template",
  "description": "",
  "snippetId": "980e46c5-b76a-183f-ffff-ffffb4770059"
}

API

/nifi-api/process-groups/ParentProcessGroupUUID/templates

Example

curl -X POST -H "Accept: application/json, text/javascript, */*; q=0.01" -H "Accept-Encoding: gzip" -H "Accept-Language: en-US,en;q=0.5" -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJz" -H "Connection: keep-alive" -H "Content-Length: 90" -H "Content-Type: application/json" -H "Host: nifi.com:9091" -H "Referer: https://nifi.com:9091/nifi/login" -H "X-Requested-With: XMLHttpRequest" -d '{"name":"My Template","description":"","snippetId":"980e46c5-b76a-183f-ffff-ffffb4770059"}' https://nifi.com:9091/nifi-api/process-groups/8c3d29aa-d1ea-1c85-ffff-ffffe6b3853a/templates | gunzip

Which returns

{
  "template": {
    "uri": "https://nifi.com:9091/nifi-api/templates/e3cad030-733e-3568-b7f5-90b2457e1637",
    "id": "e3cad030-733e-3568-b7f5-90b2457e1637",
    "groupId": "8c3d29aa-d1ea-1c85-ffff-ffffe6b3853a",
    "name": "My Template",
    "description": "",
    "timestamp": "03/05/2017 05:29:21 UTC",
    "encoding-version": "1.0"
  }
}

In the NiFi UI you will see the newly created template

[#2]

hi, i have a question

you write in the sektion Fetch Access Token

1. -d 'username=myUserName&password=YourPasswordHere' \

where i find the username and the password of nifi, while i used it without a username and password!

when i need to create one how could i do it?

[#3]

Hi, i am trying to follow your steps. unfortunately getting error, please look once

Step1: i am using to get token

curl -X POST --insecure -H \ -H "Accept: */*" \ -H "Accept-Encoding: gzip" \ -H "Accept-Language: en-US,en;q=0.5" \ -H "Connection: keep-alive" \ -H "Content-Length: 38" \ -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" \ -H "Host: https://nifitest.com/nifi" \ -H "Referer: https://nifitest.com/nifi/login" \ -H "X-Requested-With: XMLHttpRequest" \ -d 'username=XXXX&password=XXXX' \ https://nifitest.com/nifi-api/access/token | gunzip

error:

Curl: (6) Couldn't resolve host 'Accept: *' %

Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 34 0 298 --:--:-- --:--:-- --:--:-- 0 gzip: stdin: unexpected end of file

[#4]

Hi, I would like to know also about the steps on how to achieve that(manipulate the processors/pg using REST API). My own configuration is single-user(default). I tried to invoke nifi component(process groups) using HTTP GET via Postman. But when I tried to use PUT request I got 403 response status. What should I do? I've been trying to make it work (2 days) but I didn't succeed. Thanks in advance.

[#5]

@MaarufB Please make a new post with as much detail as you can around your question and use case.  This is an old topic and will not get a good response in the comments.  Feel free to @ tag me in the new post.