Created 11-29-2016 03:47 PM
Hi all,
I will put NIFI in production but i would know how to monitor processor and NIFI cluster.
Could I use API REST or something like this ?
regards
Created on 12-02-2016 03:06 PM - edited 08-19-2019 02:50 AM
There are a few ways to go about this.
Bulletin board shows all processors with WARNING and ERROR alerts and double click to actual processor. Status bar of course gives overall number of running and alerted and each processor gives metrics and history via Status History.
Reporting Tasks run in the background to provide statistical reports about what is happening in the NiFi instance. It is configured in the UI (see link below) by accessing upper right dropdown and clicking Controller Settings, and then Reporting Tasks
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks
There currently are 7 types of reporting tasks:
AmbariReportingTask | Publishes metrics from NiFi to Ambari Metrics Service (AMS). |
ControllerStatusReportingTask | Logs the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections, as well optionally logging the deltas between the previous iteration and the current iteration. |
DataDogReportingTask | Publishes metrics from NiFi to datadog. |
MonitorDiskUsage | Checks the amount of storage space available for the specified directory and warns (via a log message and a System-Level Bulletin) if the partition on which it lives exceeds some configurable threshold of storage space |
MonitorMemory | Checks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold. |
SiteToSiteProvenanceReportingTask | Publishes Provenance events using the Site To Site protocol. |
StandardGangliaReporter | Reports metrics to Ganglia so that Ganglia can be used for external monitoring of the application. |
Build your own monitoring via Rest API
https://nifi.apache.org/docs/nifi-docs/rest-api/
Created on 12-02-2016 03:06 PM - edited 08-19-2019 02:50 AM
There are a few ways to go about this.
Bulletin board shows all processors with WARNING and ERROR alerts and double click to actual processor. Status bar of course gives overall number of running and alerted and each processor gives metrics and history via Status History.
Reporting Tasks run in the background to provide statistical reports about what is happening in the NiFi instance. It is configured in the UI (see link below) by accessing upper right dropdown and clicking Controller Settings, and then Reporting Tasks
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks
There currently are 7 types of reporting tasks:
AmbariReportingTask | Publishes metrics from NiFi to Ambari Metrics Service (AMS). |
ControllerStatusReportingTask | Logs the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections, as well optionally logging the deltas between the previous iteration and the current iteration. |
DataDogReportingTask | Publishes metrics from NiFi to datadog. |
MonitorDiskUsage | Checks the amount of storage space available for the specified directory and warns (via a log message and a System-Level Bulletin) if the partition on which it lives exceeds some configurable threshold of storage space |
MonitorMemory | Checks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold. |
SiteToSiteProvenanceReportingTask | Publishes Provenance events using the Site To Site protocol. |
StandardGangliaReporter | Reports metrics to Ganglia so that Ganglia can be used for external monitoring of the application. |
Build your own monitoring via Rest API
https://nifi.apache.org/docs/nifi-docs/rest-api/
Created 03-18-2017 10:30 PM
I don't have an answer but have a related question. Is there a way to monitor details of the CPU, memory, Heap, I/O, Threads that each NiFi processor uses?
Thanks,
DR
Created 03-21-2017 03:11 PM
@Dan Rogers NiFi runs in a single JVM. As such, all Processors in the data flow will share the resources of the same JVM. These details can be monitored through the system diagnostics. They are access in the UI by clicking on the system diagnostics link in the Summary page off of the global context menu. They are loaded from the system diagnostics rest endpoint which can be accessed here:
http://localhost:8080/nifi-api/system-diagnostics
Also check out the docs [1]. Click the System Diagnostics at the bottom to expand.
Hope this helps!
[1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
Created 03-22-2018 09:19 AM
Latest Nifi Monitoring docs: (HDF 3.1.1)
Created 04-02-2018 04:21 PM
Zhen Zang,
Thanks for the updated document and information! I’m still struggling with the best way to determine how much of the JVM CPU, memory, I/O, etc. each NiFi processor is using.
Thanks,