Is there any ways if we can schedule or trigger the nifi jobs from another scheduler like run deck ?

[#1]

@Rohit Ravishankar, Any suggested answers as I have read a thread where you have been trying to schedule the nifi jobs from control M .

Thanks in advance .

[#2 · Accepted solution]

Hi @aman mittal

NiFi has it's own scheduler and provides processor level scheduling. NiFi is a flow management tools that was designed to ingest data as fast and efficient as possible. The scheduling is done in NiFi as a complete platform and at a granular level. I don't think that using an external scheduler as a general solution for NiFi flows scheduling is a good approach. However, this can make sense in some scenarios.

Here are few tips:

To implement these edges scenarios you can add the scheduling logic to your flow and trigger it from an external system through an HTTP call for instance (adding ListenHTTP or any other processor), a Kafka event or any other mechanism. In some cases, this won't be possible. I think about flows where your first processor doesn't accept incoming relationships. I am sure that there are other scenarios where this won't work too.

Also, something to look to is Wait/Notify. You can use them as a gate that you open upon receiving an external event (ex. http call from a scheduler). Here you will have two flows : one for data ingest with a blocking step with wait, and one for deblocking the flow with notify. If you have varying number of flow files to block/release, this solution can be complicated.

[#3]

@Abdelkrim Hadjidj Thanks for the solution suggestion . I believe HTTP call to wait can help my problem . I will try and post out the outcomes .

[#4]

@Abdelkrim Hadjidj , I have tried the suggested implementation but I want to trigger the ListenHTTP processor from an external input .Like I want to trigger the processor through a curl command which will trigger the mentioned nifi processor .

Is there any way we can automate this use case .

Processor should start only if we have the curl command hitting that processor .

[#5]

@aman mittal : Did you find a solution for this ?

Thanks !