Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

Solved Go to solution

Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

New Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

Hi @aman mittal

NiFi has it's own scheduler and provides processor level scheduling. NiFi is a flow management tools that was designed to ingest data as fast and efficient as possible. The scheduling is done in NiFi as a complete platform and at a granular level. I don't think that using an external scheduler as a general solution for NiFi flows scheduling is a good approach. However, this can make sense in some scenarios.

Here are few tips:

To implement these edges scenarios you can add the scheduling logic to your flow and trigger it from an external system through an HTTP call for instance (adding ListenHTTP or any other processor), a Kafka event or any other mechanism. In some cases, this won't be possible. I think about flows where your first processor doesn't accept incoming relationships. I am sure that there are other scenarios where this won't work too.

Also, something to look to is Wait/Notify. You can use them as a gate that you open upon receiving an external event (ex. http call from a scheduler). Here you will have two flows : one for data ingest with a blocking step with wait, and one for deblocking the flow with notify. If you have varying number of flow files to block/release, this solution can be complicated.

5 REPLIES 5

Re: Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

New Contributor

@Rohit Ravishankar, Any suggested answers as I have read a thread where you have been trying to schedule the nifi jobs from control M .

Thanks in advance .

Re: Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

Hi @aman mittal

NiFi has it's own scheduler and provides processor level scheduling. NiFi is a flow management tools that was designed to ingest data as fast and efficient as possible. The scheduling is done in NiFi as a complete platform and at a granular level. I don't think that using an external scheduler as a general solution for NiFi flows scheduling is a good approach. However, this can make sense in some scenarios.

Here are few tips:

To implement these edges scenarios you can add the scheduling logic to your flow and trigger it from an external system through an HTTP call for instance (adding ListenHTTP or any other processor), a Kafka event or any other mechanism. In some cases, this won't be possible. I think about flows where your first processor doesn't accept incoming relationships. I am sure that there are other scenarios where this won't work too.

Also, something to look to is Wait/Notify. You can use them as a gate that you open upon receiving an external event (ex. http call from a scheduler). Here you will have two flows : one for data ingest with a blocking step with wait, and one for deblocking the flow with notify. If you have varying number of flow files to block/release, this solution can be complicated.

Re: Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

New Contributor


@Abdelkrim Hadjidj Thanks for the solution suggestion . I believe HTTP call to wait can help my problem . I will try and post out the outcomes .

Re: Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

New Contributor

@Abdelkrim Hadjidj , I have tried the suggested implementation but I want to trigger the ListenHTTP processor from an external input .Like I want to trigger the processor through a curl command which will trigger the mentioned nifi processor .

Is there any way we can automate this use case .

Processor should start only if we have the curl command hitting that processor .

Re: Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

New Contributor

@aman mittal : Did you find a solution for this ?

Thanks !

Don't have an account?
Coming from Hortonworks? Activate your account here