Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi job control and date parameter

NiFi job control and date parameter

Rising Star

What is the most common way to kick off a daily NiFi process from a job control system or cron scheduler and pass in a $date parameter to a NiFi processor?

7 REPLIES 7

Re: NiFi job control and date parameter

Hi,

Take a look at NiFi Api, you can manipulate processors and do a lot more:

https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

to extend my answer, In your case, Start/Stop/update processor(may be from cron):

Sample API call:

#Update processor:

curl -i -X PUT -H 'Content-Type: application/json' -d '{"revision":{"version":19,"clientId":"b7aff2f1-fb0e-4abd-90f4-c64d68989a5b"},"processor":{"id":"7f033b6f-e13e-4295-85de-33444aae6c4b","parentGroupId":"02888a59-ee2b-483e-8860-b0fdf948de97","config":{"properties":{"URL":"http://finance.google.com/finance/info?client=ig&q=YHOO","Filename":"YAHOO"}}}}'  http://localhost:8080/nifi-api/controller/process-groups/02888a59-ee2b-483e-8860-b0fdf948de97/proces... 

#Start processor:

curl -i -X PUT -H 'Content-Type: application/json' -d '{"revision":{"clientId":"b7aff2f1-fb0e-4abd-90f4-c64d68989a5b"},"processGroup":{"id":"02888a59-ee2b-483e-8860-b0fdf948de97","parentGroupId":"f5ac030c-2218-414d-acef-9d1d9c6e5e84","running":"true"}}' http://localhost:8080/nifi-api/controller/process-groups/02888a59-ee2b-483e-8860-b0fdf948de97/ 

#Stop Processor:

curl -i -X PUT -H 'Content-Type: application/json' -d '{"revision":{"clientId":"b7aff2f1-fb0e-4abd-90f4-c64d68989a5b"},"processGroup":{"id":"02888a59-ee2b-483e-8860-b0fdf948de97","parentGroupId":"f5ac030c-2218-414d-acef-9d1d9c6e5e84","running":"false"}}' http://localhost:8080/nifi-api/controller/process-groups/02888a59-ee2b-483e-8860-b0fdf948de97/ 

This is one way you can get this done.

Thanks

Jobin George

Re: NiFi job control and date parameter

New Contributor

@Jobin George it is really helpful. Can you please let me know how can we find the parentID for a process group which is inside another process group?

ex : I have a process group Ingestion which has sub process groups like RDBMS pull, Mainframes pull etc.,

Also, how can we set client ID? can it be done by adding a parameter in config file ?

Re: NiFi job control and date parameter

Also note one can schedule a processor with a CRON expression directly in NiFi, can expand your options.

Re: NiFi job control and date parameter

Expert Contributor

Another option is to have your generate flow file on a re-occuring CRON schedule or timer (set to run every x hours)

Re: NiFi job control and date parameter

New Contributor

In @Jobin George, I have trouble verifying this, as I can't seem to find it in the API documentation on the 'running' attribute on the ProcessGroup.

Is there something I am missing? I need to verify that I am able to kickoff a processgroup remotely.

#Start processor:

curl -i -X PUT -H 'Content-Type: application/json'-d '{"revision":{"clientId":"b7aff2f1-fb0e-4abd-90f4-c64d68989a5b"},"processGroup":{"id":"02888a59-ee2b-483e-8860-b0fdf948de97","parentGroupId":"f5ac030c-2218-414d-acef-9d1d9c6e5e84","running":"true"}}' http://localhost:8080/nifi-api/controller/process-groups/02888a59-ee2b-483e-8860-b0fdf948de97/ 
,

Researching to see if Nifi has all the things I need in regards to a POC for an ETL project I am building out. In Jobin George's answer to start and stop processor, there is an attribute in the example...running...is this documented somewhere?

Re: NiFi job control and date parameter

@Ed S : Looks like the api has changed since I added this answer, you can find the API documentation below, under /FLOW

https://nifi.apache.org/docs/nifi-docs/rest-api/index.html 

For details on how request and response json would look like, you can click on each "Entity" of the response/request by drilling down.

To answer your question on how to start/stop Process group[with id:db8d735b-015f-1000-1c3e-974bf0ba1442],

Below command will start entire process group:

curl -i -X PUT -H 'Content-Type: application/json' -d '{"id": "db8d735b-015f-1000-1c3e-974bf0ba1442","state": "RUNNING"}' http://localhost:9090/nifi-api/flow/process-groups/db8d735b-015f-1000-1c3e-974bf0ba1442

Below command will stop entire process group:

curl -i -X PUT -H 'Content-Type: application/json' -d '{"id": "db8d735b-015f-1000-1c3e-974bf0ba1442","state": "STOPPED"}' http://localhost:9090/nifi-api/flow/process-groups/db8d735b-015f-1000-1c3e-974bf0ba1442

- Jobin

Highlighted

Re: NiFi job control and date parameter

New Contributor

@ jobin george

Is there a way to kick off multiple Nifi jobs in parallel ? For example, when it comes to sqoop we can kick off 10 different sqoop jobs at the same time in oozie, because they are all separate , I can do that, there is no restriction on that, but if I do this in Nifi, how does that work? Appreciate any help.

Don't have an account?
Coming from Hortonworks? Activate your account here