Is it possible to create NiFi workflows using REST API or any other thing?

[#1]

I'm wondering is it possible to create NiFi workflows with the existing processors using REST Api or CLI instead of creating workflows in GUI. Is REST Api can be used to create workflows or only to modify and schedule workflows. If we can create workflows using REST api or CLI, any articles or blogs to look into it?

Let's say I want to create a workflow which simply takes file form local and put it to hdfs. This can be possible by drag and drop the processors in GUI but Is it possible to do the same using REST API or CLI or any other thing which we can do programatically?  It would be a great help if you guide me regarding this.

Thanks a lot.

[#2 · Accepted solution]

@girish6 

Anything you can do through the NiFi UI can also be accomplished via the NiFi rest-api.

Within NiFi you can go to the global menu in the upper right corner and then select "Help". 
On the left side of the Help UI you will see and option in the list for the NiFi rest-api docs for the release you are running.

However, it is easier to get use to NiFi rest-api through the developer tools available in your browser.
You can open the developer tools while you are in the NiFi UI and see the rest-api calls as they are made when you perform an action on the GUI (like adding a processor component).  You can even right click on the rest-api call in the developers tool window and copy as curl to get the exact rest-api call you could make from a command line prompt.

Hope this helps you,

Matt

[#3]

Thank you so much @MattWho 

But another question popped up here. Is there any possibility to execute the xml file somewhere (not on GUI) where it can reflects in the GUI same as curl command.
Through  REST API calls we can create and modify workflows, thats totally fine. In the same way, can we create and modify workflows in the xml file and executing somewhere reflects in the GUI?

[#4]

@girish6 

By xml file, I am assuming you are referring to the flow.xml.gz file?

NiFi will only read the flow.xml.gz file during NiFi startup.  While NiFi is running the flow.xml resides completely in JVM heap memory and updates made via the UI or rest-api calls are pushed to flow.xml.gz only.

It can be very dangerous manually editing the flow.xml.gz file and is not recommended.  Your queued FlowFiles are tightly coupled  to specific connection uuids.  While the NiFI UI/rest-api would block you from removing a connection which has queued FlowFiles, the flow.xml.gz would not offer such protections.  This could result in data loss if someone where to manually edit the flow.xml.gz file and restart NiFi.  

Also keep in mind that in a multi-node NiFi cluster each node runs with its own copy of the flow.xml.gz loaded in to memory and the UI and rest-api calls are replicated to all nodes to make sure these files stay synced across your cluster.  

Hope this helps,

Matt

[#5]

@MattWho 

cool, In terms of REST API calls, Isn't it possible to create workflows using nifi rest api?

[#6]

@girish6 

Anything you can do via the NiFi UI, can also be done using the NiFi Rest-api.  
Using the developer tools available ion most browsers is a great way to learn how the rest-api calls work.  You can have the developer tools UI open while you perform similar actions on the canvas and then capture those rest-api request directly from the developer tools (most offer an option to "save as curl").

It can be challenging to build dataflows via the UI in terms of placement.  While NiFi does not care if all your components are stacked upon one another at position 0,0, if you ever try to access the UI to modify a component that way later it would be nearly impossible as it would be one blob in the center of the canvas.

You can find the rest-api docs embedded in your NiFi installation by going to the Global Menu in the upper right corner of your NiFi UI and selecting Help.

Hope this helps,

Matt

[#7]

@MattWho 

I am also trying to create a NiFi workflow through RestAPI but I have one doubt from where I will get the clientId, I can see when I am creating a processor in request payload there is a revision inside that object clientId is there as parameters. I am not able to understand where I will get that clientId. Any help will be great.

[#8]

@vineet_harkut 

I recommend starting a new question since this one already has a resolution.  You'll get better responses that way.

But the clientId can be whatever you want to use.  For example even below would work:
"revision":{"clientId":"1","version":0}

The JSON response will have the UUID for the component you created which you would then use to modify the configuration of that component in future rest api calls.

Thank you,

Matt

[#9]

Thanks @MattWho