Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

External monitoring of Nifi Flow

External monitoring of Nifi Flow

New Contributor

Hello,

 

I'm part of a sysadmin team and we are providing Nifi enabled machines to many to our on-site development teams. We do not have control of the flows that will be developed in each one. However we would like to know if there is a flow consuming too much resources without interfering on the flow development process. Is there a way to monitor the machine to point to excessive consuming flows?

1 REPLY 1
Highlighted

Re: External monitoring of Nifi Flow

Master Guru

@MarkBia 

 

NiFi executes within a single JVM and therefore only one resource pool exists from which all dataflows get resources from.  So from a standpoint of which thread from which dataflow is consuming the buck of resources, there is no endpoint to get those details.  An updateAttribute processor running in dataflow A looks no different than an updateAttribute processor running in dataflow B from a thread standpoint.

That being said, there is some indirect information you can collect from the NiFi summary UI.

Processors report stats on Tasks/Time (5 minute stat) that can tell you if a given processor is consuming more CPU time than other components.  You can also look at "Read /Write" stats that will tell you which processors are reading and writing the most bytes of data to the NiFi content repository disk.

You can get this summary information in a couple ways:

1. Create an external script that use the NiFi Rest-api to retrieve these details and then via your script parse the large json response for the specific stats you are looking for.

2. Create a new logger in the NiFi logback.xml and utlilize the NiFi ControllerStatusReportingTask to output these stats to your new logger.   Then you can have a script parse these logs for the stats you care about.

Hope this helps you,

Matt

Don't have an account?
Coming from Hortonworks? Activate your account here