question Re: NiFi Queue threshold in Archives of Support Questions (Read Only)

NiFi Queue threshold

yahya_najjar — Mon, 21 Nov 2016 00:42:38 GMT

Hello..

is it possible to send threshold (email/SNMP) once threshold is reached to X level.

I was able to use the PutEmail after MergeContent which is not very professional...but was not able to set threshold for that ?

Thanks

Re: NiFi Queue threshold

TimothySpann — Mon, 21 Nov 2016 02:31:10 GMT

https://community.hortonworks.com/content/kbentry/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html

Where do you want the threshold? On the queue? On each processor?

Set your run schedule or Scheduling Strategy.

Re: NiFi Queue threshold

bbende — Mon, 19 Aug 2019 11:12:12 GMT

There isn't a way that I know of to send alerts based on the threshold. The threshold is used by NiFi to trigger back-pressure, which means when the threshold is reached the components right before the queue will no longer be triggered to run anymore until the queue goes below the threshold. In the next Apache NiFi release (1.1) there will be a color indicator as to whether back-pressure is occurring or not:

Re: NiFi Queue threshold

yahya_najjar — Mon, 21 Nov 2016 02:51:18 GMT

Thanks Timothy,

Well, before PutSQL, incase there is a DB connection issue and and queue reaches to X level, I want to recieve some kind of alarm..

Re: NiFi Queue threshold

yahya_najjar — Mon, 21 Nov 2016 02:55:46 GMT

Thanks for your answer, is there any workaround for that ? any suggestion ? I need to trigger email/alarm incase I have connection issue (putSQL) processor ?

Re: NiFi Queue threshold

bbende — Mon, 21 Nov 2016 03:07:16 GMT

Could you use the MonitorActivity processor? If you have a constant flow of data to your database then you could connect the success relationship from PutSQL to a MonitorActivity processor, and send an alert if nothing has been successful in X minutes. It is not really tied to the queue threshold at all, but would likely indicate an error if you hadn't seen any successful flow files in a while.

You could also write a custom ReportingTask that went through all the bulletins (warnings/errors) to find bulletins for any PutSQL processor and then send an email. This requires a bit of development work though.

Re: NiFi Queue threshold

yahya_najjar — Mon, 21 Nov 2016 03:11:41 GMT

Excellent... it should do the work for me... many thanks

Re: NiFi Queue threshold

TimothySpann — Mon, 21 Nov 2016 03:20:01 GMT

https://nifi.apache.org/docs/nifi-docs/rest-api/

Could look for values: GET /flowfile-queues/{id}/listing-requests/{listing-request-id} Gets the current status of a listing request for the specified connection.

You can call that from a NiFi InvokeHTTP and check the status.

Re: NiFi Queue threshold

TimothySpann — Mon, 21 Nov 2016 03:20:32 GMT

or call the API in a script that can send a message or mail.

Re: NiFi Queue threshold

yahya_najjar — Mon, 21 Nov 2016 03:28:40 GMT

Thanks really.. didn't know this... will check it 🙂