Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

How to enable Apache NiFi metrics for JMX

Contributor

Hi Guys,
I know NiFi provides ReportTask component to export metrics to external monitoring systems.

I was wondering whether we can use a generic approach to collect NiFi metrics via JMX.

For example, there is a JMX exporter to Prometheus by collecting metrics from java application. May I use it to collect NiFi metrics to Prometheus?

Thanks.

6 REPLIES 6

Contributor

Hi @Benoit Rousseau,

Thank you for your response.

For the jmx_exporter option, my main trouble is where can I find or create the "nifi.yml".

I found a couple of template for Kafka and Cassandra. But How do I know the existing jmx metrics in NiFi.

I didn't find any documents on it.

Would you please share a template?

Thanks.

Explorer

@Alvin Jin

A basic "export all" config is as follow:

lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:
- pattern: ".*"

If you want to refine your config and set wich beans you want to fetch, these blog post might be useful

http://www.whiteboardcoder.com/2017/04/prometheus-and-jmx.html

https://blog.godatadriven.com/hbase-prometheus-monitoring

To browse beans I'd recommend using jmxterm https://github.com/jiaqi/jmxterm

su <ownerJVM> -c "$JAVA_HOME/bin/java -jar jmxterm-1.0.0-uber.jar "
$>jvms
jvms
<pidJVM> (m) - <JVM command line>
<pidJMXterm> ( ) - jmxterm-1.0.0-uber.jar
$> open <pidJVM>
$> beans
$> info -d <domain> -b <Mbean>
$> get -d <domain> -b <Mbean> <parameters>
$> close

Explorer

Hi,

Most handy option for JMX is https://github.com/prometheus/jmx_exporter

In bootstrap.conf set

java.arg.N=-javaagent:/path/to/prometheus/jmx_exporter/jmx_prometheus_javaagent.jar=portChoosen:/path/to/prometheus/jmx_exporter/nifi.yml"

There is a dedicated exporter: https://github.com/msiedlarek/nifi_exporter

Also useful to push events to Push Gateway (business view) https://github.com/mkjoerg/nifi-prometheus-reporter

Rgds,

@Alvin Jin @Benoit Rousseau

I am trying to monitor NIFI using JMX. Can you please provide steps to enable JMX for HDF Ambari based Nifi cluster?

Thanks

Explorer

hi @Srini K,

Here is the most simple process using JMX prometheus exporter in oder to store collected JMX data:

1/ download and copy jmx_exporter_jar (highly recommend version 3+) on all NiFi nodes and grant access to NiFi user

2/ create a config file, let's name it nifi.yml and copy inside at least an "export everything" like this one:

lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:- pattern: ".*"

3/ update NiFi bootstrap.conf in Ambari by adding (replace N by an available number and choose any available port) and then restart NiFi node

java.arg.N=-javaagent:/<path_to_jmx_exporter>/jmx_prometheus_javaagent.jar=<port>:/<path_to_jmx_exporter>/nifi.yml"

4/ configure prometheus server to poll nodes on above port

5/ for visualisation i'd recommend Grafana and customise an existing Dashboard like https://grafana.com/dashboards/3066

Thanks @Benoit Rousseau

I followed your steps and restarted NiFi service. NiFi Service is getting failed with below error:

ERROR [NiFi logging handler] org.apache.nifi.StdErr Error: Could not find or load main class "-javaagent:.root.software.jmx_prometheus_javaagent-0.3.1.jar=7071:.root.software.nifi.yml"