OBJECTIVE:
Add more useful service-level metrics for NiFi in Cloudera Flow Management (CFM)/Cloudera DataFlow (CDF).
OVERVIEW:
The default install of early releases of Cloudera Flow Management (CFM) includes only a few basic metrics for NiFi in the Cloudera Manager dashboards.
This article explains how to add several more service and host level metrics that improve the performance monitoring in Cloudera Manager 5.x or 6.x.
PREREQUISITES:
CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later
ADDING NiFi METRICS TO CM
Basic Metrics
1. From the NiFi status page in Cloudera Manager, select “Chart Builder” from the “Charts” menu.
2. Insert one of the following SELECT statements in the query text box:
NiFi Memory
SELECT total_mem_rss_across_nifi_nodes, total_mem_virtual_across_nifi_nodes WHERE category = SERVICE and entityName = $SERVICENAME
CPU Rates
SELECT cpu_user_rate_across_nifi_nodes, cpu_system_rate_across_nifi_nodes WHERE entityName = $SERVICENAME AND category = SERVICE
NiFi Bytes Written
SELECT write_bytes_rate_across_nifi_nodes, avg(write_bytes_rate_across_nifi_nodes) WHERE entityName = $SERVICENAME AND category = SERVICE
3. Select a chart type (“Line” works well for fluid metrics like memory and CPU)
4. Click “Build Chart”
5. Update the Title (e.g., “CPU Rates”)
6. Select “All Combined” to place all selected metrics on the same chart. Select “All Separate” to add a separate chart for each metric. (NOTE: You may see a syntax error related to the $SERVICENAME variable not being available in the builder outside of a dashboard. This may be ignored and will resolve after saving)
7. Click “Save” and select the NiFi Status Page dashboard (listed under CDH5 or CDH6, depending upon release)
8. Open the NiFi Status Page and confirm that your new metric is added
Stacked Metrics:
For some metrics, such as memory utilization, a stacked presentation better represents total usage across the service.
1. For stacked metrics, add the SELECT statement as before, for example:
CPU Rates Stacked
SELECT cpu_user_rate_across_nifi_nodes, cpu_system_rate_across_nifi_nodes WHERE entityName = $SERVICENAME AND category = SERVICE
2. Select “All Combined”
3. Select a stacked format, such as “Stack Area”
4. Build the chart
5. Save the chart to the appropriate dashboard as above
Conclusion:
Once you have added the metrics, experiment with other metric categories, filters, and aggregation to develop a dashboard that suits their needs. For additional metrics reporting capabilities, consider adding a Reporting Controller for DataDog or AppDynamics to push NiFi metrics to one of these general purpose SEIM/Operations tools.
