Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to enable Apache NiFi metrics for JMX

How to enable Apache NiFi metrics for JMX

Contributor

Hi Guys,
I know NiFi provides ReportTask component to export metrics to external monitoring systems.

I was wondering whether we can use a generic approach to collect NiFi metrics via JMX.

For example, there is a JMX exporter to Prometheus by collecting metrics from java application. May I use it to collect NiFi metrics to Prometheus?

Thanks.

6 REPLIES 6

Re: How to enable Apache NiFi metrics for JMX

Contributor

Hi @Benoit Rousseau,

Thank you for your response.

For the jmx_exporter option, my main trouble is where can I find or create the "nifi.yml".

I found a couple of template for Kafka and Cassandra. But How do I know the existing jmx metrics in NiFi.

I didn't find any documents on it.

Would you please share a template?

Thanks.

Re: How to enable Apache NiFi metrics for JMX

New Contributor

@Alvin Jin

A basic "export all" config is as follow:

lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:
- pattern: ".*"

If you want to refine your config and set wich beans you want to fetch, these blog post might be useful

http://www.whiteboardcoder.com/2017/04/prometheus-and-jmx.html

https://blog.godatadriven.com/hbase-prometheus-monitoring

To browse beans I'd recommend using jmxterm https://github.com/jiaqi/jmxterm

su <ownerJVM> -c "$JAVA_HOME/bin/java -jar jmxterm-1.0.0-uber.jar "
$>jvms
jvms
<pidJVM> (m) - <JVM command line>
<pidJMXterm> ( ) - jmxterm-1.0.0-uber.jar
$> open <pidJVM>
$> beans
$> info -d <domain> -b <Mbean>
$> get -d <domain> -b <Mbean> <parameters>
$> close

Re: How to enable Apache NiFi metrics for JMX

New Contributor

Hi,

Most handy option for JMX is https://github.com/prometheus/jmx_exporter

In bootstrap.conf set

java.arg.N=-javaagent:/path/to/prometheus/jmx_exporter/jmx_prometheus_javaagent.jar=portChoosen:/path/to/prometheus/jmx_exporter/nifi.yml"

There is a dedicated exporter: https://github.com/msiedlarek/nifi_exporter

Also useful to push events to Push Gateway (business view) https://github.com/mkjoerg/nifi-prometheus-reporter

Rgds,

Re: How to enable Apache NiFi metrics for JMX

New Contributor

@Alvin Jin @Benoit Rousseau

I am trying to monitor NIFI using JMX. Can you please provide steps to enable JMX for HDF Ambari based Nifi cluster?

Thanks

Re: How to enable Apache NiFi metrics for JMX

New Contributor

hi @Srini K,

Here is the most simple process using JMX prometheus exporter in oder to store collected JMX data:

1/ download and copy jmx_exporter_jar (highly recommend version 3+) on all NiFi nodes and grant access to NiFi user

2/ create a config file, let's name it nifi.yml and copy inside at least an "export everything" like this one:

lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:- pattern: ".*"

3/ update NiFi bootstrap.conf in Ambari by adding (replace N by an available number and choose any available port) and then restart NiFi node

java.arg.N=-javaagent:/<path_to_jmx_exporter>/jmx_prometheus_javaagent.jar=<port>:/<path_to_jmx_exporter>/nifi.yml"

4/ configure prometheus server to poll nodes on above port

5/ for visualisation i'd recommend Grafana and customise an existing Dashboard like https://grafana.com/dashboards/3066

Re: How to enable Apache NiFi metrics for JMX

New Contributor

Thanks @Benoit Rousseau

I followed your steps and restarted NiFi service. NiFi Service is getting failed with below error:

ERROR [NiFi logging handler] org.apache.nifi.StdErr Error: Could not find or load main class "-javaagent:.root.software.jmx_prometheus_javaagent-0.3.1.jar=7071:.root.software.nifi.yml"