Monitoring nifi queue and trigger an email when queue is 80% full

Question

Good Day!We have a requirement to read the TCP port data and publish it to Kafka. But at times the Kafka gets into a hung state and we need to restart the processor manually. So please help us with a way to configure the "Put Email" processor to trigger an email whenever this queue gets 80% full. Thanks in advance!

MattWho · Answer

@NagendraKumar

The image you shared indicates that your PublishKafka processor is producing a bulletin.
What is the nature of the exception being reported in that bulletin?
I also see what appears to be only one connection exiting your PublishKafka processors. The publishKafka processor has multiple relationships. Are you auto-terminating the failure" relationship? If so, i never recommend doing that.

-----------
Now when it comes to monitoring queue sizes/thresholds, you could use the ControllerStatusReportingTask NiFi reporting task to output these stats for all connections to a dedicated logs (see Additional Details... for how to setup dedicated logs via NiFi's logback.xml). You can then create a NiFi dataflow that tails the dedicated connection log and parses the ingested log entries for connections where thresholds exceed your 80% threshold and route those to a putEmail processor for yoru notification needs.

(preferred) Another option here is to use the SiteToSiteStatusReportingTask to report specifically on NiFi connections and feed that data in to a dataflow that parses for threshold in excess of 80% and route those to a putEmail processor. This method has less overhead as it does not write to a NiFi log, require tailing logs, can be setup to report on connections, and reports in a more structured format (see Additional Details... ).

----------
From your image I can also see your PublishKafka processor reporting 3 actively running threads. You mentioned the processor becomes hung? Have you analyzed a series of thread dumps to identify where it is getting hung?
NiFi also offers a way to monitor for long running tasks:
Runtime Monitoring Properties

You could use this in conjunction with the SiteToSiteBulletinReportingTask to construct a dataflow that could send an email alert when task are detected on processor that have been running in excess of the configured threshold. This runtime monitoring does have an impact on your overall NiFi performance due to the overhead needed to run it. So if you find it impacts your throughput performance negatively, you'll need to stop using it.

__________

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.