Skip to main content
TimothySpann
Platinum ⭐️⭐️⭐️
TimothySpann
Platinum ⭐️⭐️⭐️
April 3, 2017

Monitor Apache NiFi with Apache NiFi

  • October 5, 2026 at 8:37 AM
  • 5 replies
  • 17770 views
  • Translate All

Monitoring Apache NiFi

14305-reportingtasktoganglia.png

14304-reportingtasks.png

It's really important to pick some Reporting Tasks to let you know what's happening in Apache NiFi servers. Ambari will send it to your HDF Ambari which will show the results in nice Grafana graphs, charts and tables.

You can also monitor disk usage, memory and also send tasks to DataDog, Ganglia and Other Servers. It's also easy to write your own Reporting Task if you need a different one.

14306-monitoringflowspluschatops.png

14303-monitoractivitytoslackops.png

One of the ways to monitor your Apache NiFi Data Flows is to use the MonitorActivity processor which will create messages that can be sent to your Operations Dashboard, Console or elsewhere.

For people doing ChatOps, you can easily push these messages to Slack (there's a processor for that) PutSlack. You could also send a REST call to HipChat or other chat tools. Pretty easy to wrap that up in a custom processor as well.

Other Things to Monitor

REST END Points

server:port/nifi-api/system-diagnostics

See: https://nifi.apache.org/docs/nifi-docs/rest-api/

Logs

...nifi/logs/nifi-app.log and

..nifi/logs/nifi-user.log

These can be ingested with Apache NiFi for detailed log processing.

You can filter and send some messages to SumoLogic or elsewhere via Apache NiFi.

See: https://community.hortonworks.com/content/kbentry/67309/routing-logs-through-apache-nifi-to-phoenix-hdfs-a.html

    5 replies

    scarroll
    scarroll
    Gold ⭐️
    April 3, 2017

    PutSlack was such a good addition!

    Be careful ingesting nifi-app.log though! I've tried this before and it quickly spirals out of control as each read of the log also generates log entries which then get picked up and generate more log entries.

    Translate
    Raj_B
    Raj_B
    Gold ⭐️
    April 4, 2017

    @Sebastian Carroll IMHO you would need a separate NiFi instance for that purpose, same goes if you want to archive Provenance events from the NiFi instance; another option would be to send the logs to Splunk, etc. for log processing and for any analytics on top of that (dashboard, alerts, etc.)

    Translate
    TimothySpann
    Platinum ⭐️⭐️⭐️
    TimothySpannAuthor
    Platinum ⭐️⭐️⭐️
    March 14, 2018

    Yes, much safer to have another instance you can use for reporting and such. Even if it's just one node.

    Translate
    mrstarc
    mrstarc
    Silver ⭐️
    August 20, 2019
    As a general best practice, I suggest sending those metrics to an all-together separate monitoring system (something like InfluxDB). You can’t effectively monitor a thingy with the same thing. If that thingy fails… you risk losing visibility. #JustSayin
     
     
    Translate
    hckorkmaz01
    hckorkmaz01
    Silver ⭐️⭐️
    November 21, 2025

    Hi everyone, I hope you’re doing well. I am working on a dataflow in Apache NiFi 1.18, and I need to retrieve the queue size information (flowfile count and content size) directly within NiFi itself, not via an external script or Postman. I know that the NiFi REST API provides this data, and I can access it successfully using external tools. However, my goal is to access queue metrics from inside NiFi, for example through processors like InvokeScriptedProcessor, QueryNiFiReportingTask, or any other built-in mechanism, without sending an external REST API request from outside NiFi. Is there a recommended approach, processor, or reporting task that allows NiFi to read its own queue sizes internally? If not, what would be the best practice to achieve this? Any guidance or examples would be greatly appreciated. Thank you in advance!

     
     
    Translate
    Terms & ConditionsAccessibility statement