Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to autmatically terminate and restart a hung-up processor after x minutes

avatar
New Contributor

Hi,

I'm using a lot of flows with NiFi 1.19 (Java 11, Win 64bit)

I've got problems with a 3rd-party-processor (out-of-support): every few days this processor hung up, so I have to stop it, select "terminate" and start the processor new.

Understanding NiFi's "Terminate" option on running components. 

Is there any possibilty to configure a processor, so that it is terminated and restartet automatically if it runs longer than x minutes?

Frank

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Frank37 

Short answer is "No".  There is nothing built within the NiDi core controller to accomplish exactly what you are trying to do.

NiFi provides the ability to enable a long running task monitor which will alert to long running component threads, but that is all it does.  This new feature was introduced in Apache NiFi 1.14 and disabled by default later in NIFI-8645 because it has an impact on performance because it needs to take thread dumps periodically and is only intended for troubleshooting.

NiFi component processors are just pluggable components.  NiFi core handles allocation of thread(s) to a component at component scheduled execution, but after a thread has been given to the processor, it is that processors code that must handle the lifespan of the thread.  NiFi core would have no way of knowing how long a thread should or should not take to execute per every processor you add to the canvas.  

As you know from the linked article, the thread if truly hung will remain hung until NiFi is restarted, terminate will not kill a thread. So if NiFi was to imply an auto-terminate on all thread taking longer then X, that could be dangerous,  Might end up terminating threads that actually do take longer, could end up with a continuously incrementing of hung threads by a misbehaving processor hat goes unnoticed, etc...  Also once a thread has been terminated the NiFi core scheduler no longer cares about it, so starting processor again simply allows that processor to get a new thread from the cores Max Timer Driven thread pool.

Best is to figure out why the threads are being hung and address it through processor configuration or code changes as needed.

That being said, anything you can do through the UI can also be accomplished via the NiFi rest-api. This means you could possibly add a MonitorActivity processor after this custom processor that could use a lack of activity generated FlowFile to trigger a dataflow that makes rest-api calls to stop, terminate, and start the processor (but i would not recommend this approach for reason mentioned earlier).  But it is not a simple dataflow (involves making request to change processor state stopped, then checking status of processor to see if it stopped or is stopping (where stopping indicates it is waiting on active threads), if it is "stopping" make a request to terminate thread, and finally restart processor).    This also assumes a continuous flow fo FlowFiles through he processor.  If it gets FlowFiles sporadicly, this would not work well.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

View solution in original post

3 REPLIES 3

avatar
Community Manager

@Frank37 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @SAMSAL  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Master Mentor

@Frank37 

Short answer is "No".  There is nothing built within the NiDi core controller to accomplish exactly what you are trying to do.

NiFi provides the ability to enable a long running task monitor which will alert to long running component threads, but that is all it does.  This new feature was introduced in Apache NiFi 1.14 and disabled by default later in NIFI-8645 because it has an impact on performance because it needs to take thread dumps periodically and is only intended for troubleshooting.

NiFi component processors are just pluggable components.  NiFi core handles allocation of thread(s) to a component at component scheduled execution, but after a thread has been given to the processor, it is that processors code that must handle the lifespan of the thread.  NiFi core would have no way of knowing how long a thread should or should not take to execute per every processor you add to the canvas.  

As you know from the linked article, the thread if truly hung will remain hung until NiFi is restarted, terminate will not kill a thread. So if NiFi was to imply an auto-terminate on all thread taking longer then X, that could be dangerous,  Might end up terminating threads that actually do take longer, could end up with a continuously incrementing of hung threads by a misbehaving processor hat goes unnoticed, etc...  Also once a thread has been terminated the NiFi core scheduler no longer cares about it, so starting processor again simply allows that processor to get a new thread from the cores Max Timer Driven thread pool.

Best is to figure out why the threads are being hung and address it through processor configuration or code changes as needed.

That being said, anything you can do through the UI can also be accomplished via the NiFi rest-api. This means you could possibly add a MonitorActivity processor after this custom processor that could use a lack of activity generated FlowFile to trigger a dataflow that makes rest-api calls to stop, terminate, and start the processor (but i would not recommend this approach for reason mentioned earlier).  But it is not a simple dataflow (involves making request to change processor state stopped, then checking status of processor to see if it stopped or is stopping (where stopping indicates it is waiting on active threads), if it is "stopping" make a request to terminate thread, and finally restart processor).    This also assumes a continuous flow fo FlowFiles through he processor.  If it gets FlowFiles sporadicly, this would not work well.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

avatar
Community Manager

@Frank37 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community: