Apache NiFi Queue Size

[#1]

Hi everyone, I hope you’re doing well. I am working on a dataflow in Apache NiFi 1.18, and I need to retrieve the queue size information (flowfile count and content size) directly within NiFi itself, not via an external script or Postman. I know that the NiFi REST API provides this data, and I can access it successfully using external tools. However, my goal is to access queue metrics from inside NiFi, for example through processors like InvokeScriptedProcessor, QueryNiFiReportingTask, or any other built-in mechanism, without sending an external REST API request from outside NiFi. Is there a recommended approach, processor, or reporting task that allows NiFi to read its own queue sizes internally? If not, what would be the best practice to achieve this? Any guidance or examples would be greatly appreciated. Thank you in advance!

[#2]

@hckorkmaz01, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @MattWho, who may be able to assist you further.

Please feel free to provide any additional information or details about your query. We hope that you will find a satisfactory solution to your question.

[#3]

@hckorkmaz01 

There are a few options here:

1. SiteToSiteStatusReportingTask - Utilizes NiFi's Site-To-Site protocol to send Records to a NiFi Remote Input Port.  It can be configured to send specific component type metrics (Connection) in your case.   The Site-To-Site reporting tasks function as task specific remote process groups for generating and sending FlowFiles to the configured target Remote Input port.
2. InvokeHTTP processor - Can be used to invoke the rest-api endpoint to get the metrics you want directly in a NiFi dataflow.

Site-To-Site reference documentation:
Admin guide: Site-To-Site Properties (core setup)
User guide: Site-To-Site (user guide)

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Hi,

Apologies for asking this here, as a new member of the forum I want to post new questions but don't seem to have the ability to do so.  Can someone please explain how I can do this as I want to ask some technical questions about nifi, in particular API queries for provenance.

Thank you.

[#5]

@zzzz77 

To ask a community question:
1. Go to https://community.cloudera.com/
2. Make sure you are logged in
3. Click on the "Ask a Question" button:

4. Fill out all form:

Note: Adding "labels" make your question more visible to those community members that specialize in specific products.

Hope this helps,

Matt

[#6]

Hi,
Can this be done with PrometheusReportingTask?
Thank you
Halil Can KORKMAZ

[#7]

@hckorkmaz01 

While you are currently still using Apache NiFi 1.x major release version, it has reached end of life and is no longer receiving contributions.  As such components will not get library updates or security fixes going forward.  

Apache NiFi 2.x is currently active major release being contributed to in the community.  The PrometheusReportingTask was deprecated in Apache NiFi 1.x and officially removed in Apache NiFi 2.x major release.   So I would avoid using it as you will eventually need to move to Apache NiFi 2.x to maintain a secure supported product release.  But technically, this reporting task, while not well maintained in the community, is capable of creating a prometheus endpoint which exposes metrics for all components (includes connections) for consumption.

That being said, Cloudera has taken steps to create Cloudera versions of many of the deprecated and removed components in Apache NiFi 2.x; as well as, introduced many components not available at all in any Apache release version (PrometheusReportingTask is not one of them that was retained).
https://docs.cloudera.com/cfm/4.11.0/nifi-components-cfm/components/


NOTE:  You are already using a considerably older Apache NiFI 1.18 release.  Many bug fixes and CVEs security issues have been addressed since that release.  If you cannot yet move to Apache NiFi 2.x, you should at least be on the most recent release of Apache NIFi 1.28.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt