NiFi Queue threshold

[#1]

Hello..

is it possible to send threshold (email/SNMP) once threshold is reached to X level.

I was able to use the PutEmail after MergeContent which is not very professional...but was not able to set threshold for that ?

Thanks

[#2]

https://community.hortonworks.com/content/kbentry/7882/hdfnifi-best-practices-for-setting-up-a-high-...

Where do you want the threshold? On the queue? On each processor?

Set your run schedule or Scheduling Strategy.

[#3]

Thanks Timothy,

Well, before PutSQL, incase there is a DB connection issue and and queue reaches to X level, I want to recieve some kind of alarm..

[#4]

https://nifi.apache.org/docs/nifi-docs/rest-api/

Could look for values: GET /flowfile-queues/{id}/listing-requests/{listing-request-id} Gets the current status of a listing request for the specified connection.

You can call that from a NiFi InvokeHTTP and check the status.

[#5]

or call the API in a script that can send a message or mail.

[#6]

Thanks really.. didn't know this... will check it 🙂

[#7 · Accepted solution]

There isn't a way that I know of to send alerts based on the threshold. The threshold is used by NiFi to trigger back-pressure, which means when the threshold is reached the components right before the queue will no longer be triggered to run anymore until the queue goes below the threshold. In the next Apache NiFi release (1.1) there will be a color indicator as to whether back-pressure is occurring or not:

[#8]

Thanks for your answer, is there any workaround for that ? any suggestion ? I need to trigger email/alarm incase I have connection issue (putSQL) processor ?

[#9]

Could you use the MonitorActivity processor? If you have a constant flow of data to your database then you could connect the success relationship from PutSQL to a MonitorActivity processor, and send an alert if nothing has been successful in X minutes. It is not really tied to the queue threshold at all, but would likely indicate an error if you hadn't seen any successful flow files in a while.

You could also write a custom ReportingTask that went through all the bulletins (warnings/errors) to find bulletins for any PutSQL processor and then send an email. This requires a bit of development work though.

[#10]

Excellent... it should do the work for me... many thanks