Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Rising Star

OBJECTIVE:

Add more useful service-level metrics for NiFi in Cloudera Flow Management (CFM)/Cloudera DataFlow (CDF).

 

OVERVIEW:

The default install of early releases of Cloudera Flow Management (CFM) includes only a few basic metrics for NiFi in the Cloudera Manager dashboards.

 

This article explains how to add several more service and host level metrics that improve the performance monitoring in Cloudera Manager 5.x or 6.x.

 

PREREQUISITES:

CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later

 

ADDING NiFi METRICS TO CM

 

Basic Metrics

1. From the NiFi status page in Cloudera Manager, select “Chart Builder” from the “Charts” menu.

2. Insert one of the following SELECT statements in the query text box:

 

NiFi Memory

 

SELECT total_mem_rss_across_nifi_nodes, total_mem_virtual_across_nifi_nodes WHERE category = SERVICE and entityName = $SERVICENAME

 

 

CPU Rates

 

SELECT cpu_user_rate_across_nifi_nodes, cpu_system_rate_across_nifi_nodes  WHERE entityName = $SERVICENAME AND category = SERVICE

 

 

NiFi Bytes Written

 

SELECT write_bytes_rate_across_nifi_nodes, avg(write_bytes_rate_across_nifi_nodes)  WHERE entityName = $SERVICENAME AND category = SERVICE

 

 

3. Select a chart type (“Line” works well for fluid metrics like memory and CPU)

4. Click “Build Chart”

5. Update the Title (e.g., “CPU Rates”)

6. Select “All Combined” to place all selected metrics on the same chart. Select “All Separate” to add a separate chart for each metric. (NOTE: You may see a syntax error related to the $SERVICENAME variable not being available in the builder outside of a dashboard. This may be ignored and will resolve after saving)

 

Screenshot of new chart before savingScreenshot of new chart before saving

 

7. Click “Save” and select the NiFi Status Page dashboard (listed under CDH5 or CDH6, depending upon release)
Screenshot of CM chart save operationScreenshot of CM chart save operation

 

8. Open the NiFi Status Page and confirm that your new metric is added
Screenshot with new NiFi metrics charts addedScreenshot with new NiFi metrics charts added

 

Stacked Metrics:

For some metrics, such as memory utilization, a stacked presentation better represents total usage across the service.

 

1. For stacked metrics, add the SELECT statement as before, for example:

 
CPU Rates Stacked

 

SELECT cpu_user_rate_across_nifi_nodes, cpu_system_rate_across_nifi_nodes  WHERE entityName = $SERVICENAME AND category = SERVICE

 

 

2. Select “All Combined”

3. Select a stacked format, such as “Stack Area”

4. Build the chart


CM chart builder, stacked area memoryCM chart builder, stacked area memory

 

5. Save the chart to the appropriate dashboard as above

 

Conclusion:

Once you have added the metrics, experiment with other metric categories, filters, and aggregation to develop a dashboard that suits their needs. For additional metrics reporting capabilities, consider adding a Reporting Controller for DataDog or AppDynamics to push NiFi metrics to one of these general purpose SEIM/Operations tools.

 

621 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
6 of 6
Last update:
‎10-04-2019 01:46 PM
Updated by:
 
Contributors
Top Kudoed Authors