NIFI : Monitoring processor and nifi Service

[#1]

Hi all,

I will put NIFI in production but i would know how to monitor processor and NIFI cluster.

Could I use API REST or something like this ?

regards

[#2 · Accepted solution]

There are a few ways to go about this.

1. Native monitoring

Bulletin board shows all processors with WARNING and ERROR alerts and double click to actual processor. Status bar of course gives overall number of running and alerted and each processor gives metrics and history via Status History.

https://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_HDF_GettingStarted/content/monitoring-nifi...

2. Reporting Tasks (Push)

Reporting Tasks run in the background to provide statistical reports about what is happening in the NiFi instance. It is configured in the UI (see link below) by accessing upper right dropdown and clicking Controller Settings, and then Reporting Tasks

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks

There currently are 7 types of reporting tasks:

AmbariReportingTask	Publishes metrics from NiFi to Ambari Metrics Service (AMS).
ControllerStatusReportingTask	Logs the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections, as well optionally logging the deltas between the previous iteration and the current iteration.
DataDogReportingTask	Publishes metrics from NiFi to datadog.
MonitorDiskUsage	Checks the amount of storage space available for the specified directory and warns (via a log message and a System-Level Bulletin) if the partition on which it lives exceeds some configurable threshold of storage space
MonitorMemory	Checks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold.
SiteToSiteProvenanceReportingTask	Publishes Provenance events using the Site To Site protocol.
StandardGangliaReporter	Reports metrics to Ganglia so that Ganglia can be used for external monitoring of the application.

3. Rest API (Pull)

Build your own monitoring via Rest API

https://nifi.apache.org/docs/nifi-docs/rest-api/

4. Misc

• MonitorActivity Processor: Indicate when flow has not recieved data after a specified period of time, and again when the activity is restored https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MonitorActivit...
• Dedicated process groups to send alerts: http://ec2-52-89-10-2.us-west-2.compute.amazonaws.com/questions/54135/apache-nifi-how-to-productioni...
• Custom Logging: https://community.hortonworks.com/articles/65027/nifi-easy-custom-logging-of-diverse-sources-in-mer....

[#3]

I don't have an answer but have a related question. Is there a way to monitor details of the CPU, memory, Heap, I/O, Threads that each NiFi processor uses?

Thanks,

DR

[#4]

@Dan Rogers NiFi runs in a single JVM. As such, all Processors in the data flow will share the resources of the same JVM. These details can be monitored through the system diagnostics. They are access in the UI by clicking on the system diagnostics link in the Summary page off of the global context menu. They are loaded from the system diagnostics rest endpoint which can be accessed here:

http://localhost:8080/nifi-api/system-diagnostics

Also check out the docs [1]. Click the System Diagnostics at the bottom to expand.

Hope this helps!

[1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

[#5]

Latest Nifi Monitoring docs: (HDF 3.1.1)

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_getting-started-with-apache-nifi/content...

[#6]

Zhen Zang,

Thanks for the updated document and information! I’m still struggling with the best way to determine how much of the JVM CPU, memory, I/O, etc. each NiFi processor is using.

Thanks,