Gold ⭐️

Solved

how create, export and import full nifi template by nifi api

8 replies
19999 views
Translate All

hi cloudera community,

creating a full template using the nifi webui mode works perfectly

we need to create the template, export and import using the nifi api, has anyone managed to configure this?

yagoaparecidotiAnswer

Gold ⭐️

hi @MattWho,

It was this process of creating the "template" in XML that I was referring to in how to do it through the nifi API

but, since it has been discontinued, I will not continue.

thanks for the information.

vaishaakb

Gold ⭐️⭐️⭐️

Hi @yagoaparecidoti Thank you for bringing this to our attention.

Did you get a chance to review these? if not, Let me help you here:

How to Create a Nifi Template using REST API:
https://community.cloudera.com/t5/Community-Articles/Creating-NiFi-Template-via-Rest-API/ta-p/244624

How to take a backup of NiFi using REST API:
https://community.cloudera.com/t5/Support-Questions/NIFI-Create-Backup-Template-with-NIFI-API/m-p/124467#M87211

How to Import Template using API:
You may refer the overview here: https://github.com/aperepel/nifi-api-deploy

NiFi API Ref:
https://nifi.apache.org/docs/nifi-docs/rest-api/index.html#:~:text=Delete%20a%20snippet-,templates,-Download%20a%20template

Let us know how it goes.

Cheers
V

Translate

MattWho

Titanium ⭐️⭐️⭐️

@yagoaparecidoti

NiFi templates have been deprecated in Apache NiFi. NiFi flow definitions are the replacement.

The ability to create and import templates no longer exists as of Apache NiFi 2.x releases.
NiFi templates as well as NiFi flow definitions exist in the Apache NiFi 1.x releases.

The first question is what do you mean by "full template"?

One of the best ways to navigate the rest-api calls needed to accomplish any task is through the use of the developer tools available in your browser.

Open the developer tools and execute the steps via the NiFi UI to accomplish your use case. With each step you can capture the rest-api requests that are being made. Developer tools will even allow you right click on request and select "copy as curl". This will allow you to see the rest-api endpoint and the added headers and raw-data that may go with the request. The browser will add numerous additional headers that are not needed.

@vaishaakb Has provided some other community articles that are a good resources.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

yagoaparecidotiAuthor

Gold ⭐️

hi @vaishaakb

thanks for the links, I'm checking to see if it's what I need.

@MattWho

when I refer to "full template" I would mean creating the "nifi flow" template. This creation can be done via webui by clicking on the "create template" option within "nifi flow" and then downloading this template in json format and importing this template to another nifi

Translate

MattWho

Titanium ⭐️⭐️⭐️

@yagoaparecidoti
I think there is still confusion here on what you are really doing.
Templates are in XML format and not JSON. There is not option to download a template in JSON format.

Templates and Flow Definitions are two different things (templates are deprecated and fully removed as of Apache NiFi 2)

In Apache NiFi 0.x and 1.x versions you can write click on the canvas or on a selection of components and select "create template" from context menu displayed or by clicking teh create template option from the operate panel

This results in the creation of an XML template that you need to navigate to the template UI under the NiFi global menu in order to download the XML template.

Flow Definitions are in Json format and can be created by right clicking on process group or the canvas of the process group and selecting "Download Flow Definition". This prompts the creation of flow definition json you store outside of your NiFi. A Flow i Json is uploaded to NiFi by dragging the Process Group icon to the canvas and clicking on the browse icon to select your json for upload to the UI.

Flow definitions are NOT stored within your NiFi.

I would advise strongly against using NIFi templates as they will become unusable in the newer versions of NiFi. Templates where deprecated for two reasons:
1. They are created and held in NiFi which means they consume NiFi heap memory space until they are deleted via the template UI.
2. They use XML format. NiFI has moved away from xml based flows. The flow.xml.gz is also deprecated with the flow.json.gz format.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

yagoaparecidotiAuthorAnswer

Gold ⭐️

hi @MattWho,

It was this process of creating the "template" in XML that I was referring to in how to do it through the nifi API

but, since it has been discontinued, I will not continue.

thanks for the information.

Translate

MattWho

Titanium ⭐️⭐️⭐️

@yagoaparecidoti

There are still multiple ways to move complete dataflows from one NiFi to Another NiFi.

When you start a NiFi for the first time, NiFi create the root Process Group (PG). This is the blank canvas that you see when you log in. I would suggest creating a Child PG at the root PG level and build all you dataflows within the new child PG.

Method 1:

Use NiFi-Registry:
- You can setup a Single NiFi-Registry that all your NiFi deployments can use.
- Start Version control on your child PG. This will write the PG flow definition to that NiFi-Registry.
- Give your other NiFi instance the ability to read the NiFi-Registry Bucket and flow in which that flow definition is stored.
- On your other NiFi instances import that version controlled PG from the shared NiFi-Registry.
- Anytime a newer version of that Flow definition is pushed to NiFi-Registry, all your other NiFi instances using it will show a newer version as being available.

Method 2:

You can utilize the rest-api to download a flow definition of that child PG.
You can utilize the rest-api on other instances of NiFi to instantiate that downloaded flow definition onto the root PG of those instances.

As far as the NiFi REST-API goes, it is always easier to utilize the "developer tools" available within your browser to capture the rest-api calls as they are made when you manually perform the steps directly via the NiFi UI. This allows you to copy the rest-api call as a curl command. From that captured curl command you can see the rest-api endpoint used to perform the request along with format of any data or variables needed in the request.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

yagoaparecidotiAuthor

Gold ⭐️

hi @MattWho ,

i tried to follow the "developer tools", the collected curl commands are very "polluted", so I couldn't download flow, etc.

i will try new approaches, if I am successful I will come back again.

Translate

steven-matison

Platinum ⭐️⭐️

Below is a blog post for a flow i made using the nifi api, which will accept a template (XML), load the template, get template, then follow chain of api calls to get the appropriate flow definiton file (JSON).

https://cldr-steven-matison.github.io/blog/dataflow-xml-to-json/

The concepts you need to understand are demonstrated with invokeHttp to execute the sample calls. Check the nifi api docs. You can do these calls with curl, etc., but i chose to do them in nifi. Be mindful that to start using the api from outside of nifi, you need to be following the initial api call for an access token to be used with the rest of the calls.

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.