Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NIFI : Monitoring processor and nifi Service

avatar
Rising Star

Hi all,

I will put NIFI in production but i would know how to monitor processor and NIFI cluster.

Could I use API REST or something like this ?

regards

1 ACCEPTED SOLUTION

avatar
Guru

There are a few ways to go about this.

1. Native monitoring

Bulletin board shows all processors with WARNING and ERROR alerts and double click to actual processor. Status bar of course gives overall number of running and alerted and each processor gives metrics and history via Status History.

https://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_HDF_GettingStarted/content/monitoring-nifi...

10004-screen-shot-2016-12-02-at-94006-am.png

2. Reporting Tasks (Push)

Reporting Tasks run in the background to provide statistical reports about what is happening in the NiFi instance. It is configured in the UI (see link below) by accessing upper right dropdown and clicking Controller Settings, and then Reporting Tasks

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks

There currently are 7 types of reporting tasks:

AmbariReportingTaskPublishes metrics from NiFi to Ambari Metrics Service (AMS).
ControllerStatusReportingTaskLogs the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections, as well optionally logging the deltas between the previous iteration and the current iteration.
DataDogReportingTaskPublishes metrics from NiFi to datadog.
MonitorDiskUsageChecks the amount of storage space available for the specified directory and warns (via a log message and a System-Level Bulletin) if the partition on which it lives exceeds some configurable threshold of storage space
MonitorMemoryChecks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold.
SiteToSiteProvenanceReportingTaskPublishes Provenance events using the Site To Site protocol.
StandardGangliaReporterReports metrics to Ganglia so that Ganglia can be used for external monitoring of the application.

3. Rest API (Pull)

Build your own monitoring via Rest API

https://nifi.apache.org/docs/nifi-docs/rest-api/

4. Misc

View solution in original post

5 REPLIES 5

avatar
Guru

There are a few ways to go about this.

1. Native monitoring

Bulletin board shows all processors with WARNING and ERROR alerts and double click to actual processor. Status bar of course gives overall number of running and alerted and each processor gives metrics and history via Status History.

https://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_HDF_GettingStarted/content/monitoring-nifi...

10004-screen-shot-2016-12-02-at-94006-am.png

2. Reporting Tasks (Push)

Reporting Tasks run in the background to provide statistical reports about what is happening in the NiFi instance. It is configured in the UI (see link below) by accessing upper right dropdown and clicking Controller Settings, and then Reporting Tasks

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Reporting_Tasks

There currently are 7 types of reporting tasks:

AmbariReportingTaskPublishes metrics from NiFi to Ambari Metrics Service (AMS).
ControllerStatusReportingTaskLogs the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections, as well optionally logging the deltas between the previous iteration and the current iteration.
DataDogReportingTaskPublishes metrics from NiFi to datadog.
MonitorDiskUsageChecks the amount of storage space available for the specified directory and warns (via a log message and a System-Level Bulletin) if the partition on which it lives exceeds some configurable threshold of storage space
MonitorMemoryChecks the amount of Java Heap available in the JVM for a particular JVM Memory Pool. If the amount of space used exceeds some configurable threshold, will warn (via a log message and System-Level Bulletin) that the memory pool is exceeding this threshold.
SiteToSiteProvenanceReportingTaskPublishes Provenance events using the Site To Site protocol.
StandardGangliaReporterReports metrics to Ganglia so that Ganglia can be used for external monitoring of the application.

3. Rest API (Pull)

Build your own monitoring via Rest API

https://nifi.apache.org/docs/nifi-docs/rest-api/

4. Misc

avatar
New Contributor

I don't have an answer but have a related question. Is there a way to monitor details of the CPU, memory, Heap, I/O, Threads that each NiFi processor uses?

Thanks,

DR

avatar
Rising Star

@Dan Rogers NiFi runs in a single JVM. As such, all Processors in the data flow will share the resources of the same JVM. These details can be monitored through the system diagnostics. They are access in the UI by clicking on the system diagnostics link in the Summary page off of the global context menu. They are loaded from the system diagnostics rest endpoint which can be accessed here:

http://localhost:8080/nifi-api/system-diagnostics

Also check out the docs [1]. Click the System Diagnostics at the bottom to expand.

Hope this helps!

[1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

avatar
Expert Contributor

avatar
New Contributor

Zhen Zang,

Thanks for the updated document and information! I’m still struggling with the best way to determine how much of the JVM CPU, memory, I/O, etc. each NiFi processor is using.

Thanks,