I know NiFi provides ReportTask component to export metrics to external monitoring systems.
I was wondering whether we can use a generic approach to collect NiFi metrics via JMX.
For example, there is a JMX exporter to Prometheus by collecting metrics from java application. May I use it to collect NiFi metrics to Prometheus?
Hi @Benoit Rousseau,
Thank you for your response.
For the jmx_exporter option, my main trouble is where can I find or create the "nifi.yml".
I found a couple of template for Kafka and Cassandra. But How do I know the existing jmx metrics in NiFi.
I didn't find any documents on it.
Would you please share a template?
A basic "export all" config is as follow:
lowercaseOutputLabelNames: true lowercaseOutputName: true rules: - pattern: ".*"
If you want to refine your config and set wich beans you want to fetch, these blog post might be useful
To browse beans I'd recommend using jmxterm https://github.com/jiaqi/jmxterm
su <ownerJVM> -c "$JAVA_HOME/bin/java -jar jmxterm-1.0.0-uber.jar " $>jvms jvms <pidJVM> (m) - <JVM command line> <pidJMXterm> ( ) - jmxterm-1.0.0-uber.jar $> open <pidJVM> $> beans $> info -d <domain> -b <Mbean> $> get -d <domain> -b <Mbean> <parameters> $> close
Most handy option for JMX is https://github.com/prometheus/jmx_exporter
In bootstrap.conf set
There is a dedicated exporter: https://github.com/msiedlarek/nifi_exporter
Also useful to push events to Push Gateway (business view) https://github.com/mkjoerg/nifi-prometheus-reporter
hi @Srini K,
Here is the most simple process using JMX prometheus exporter in oder to store collected JMX data:
1/ download and copy jmx_exporter_jar (highly recommend version 3+) on all NiFi nodes and grant access to NiFi user
2/ create a config file, let's name it nifi.yml and copy inside at least an "export everything" like this one:
lowercaseOutputLabelNames: true lowercaseOutputName: true rules:- pattern: ".*"
3/ update NiFi bootstrap.conf in Ambari by adding (replace N by an available number and choose any available port) and then restart NiFi node
4/ configure prometheus server to poll nodes on above port
5/ for visualisation i'd recommend Grafana and customise an existing Dashboard like https://grafana.com/dashboards/3066
Thanks @Benoit Rousseau
I followed your steps and restarted NiFi service. NiFi Service is getting failed with below error:
ERROR [NiFi logging handler] org.apache.nifi.StdErr Error: Could not find or load main class "-javaagent:.root.software.jmx_prometheus_javaagent-0.3.1.jar=7071:.root.software.nifi.yml"