Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi queue monitoring

avatar
Expert Contributor

Hi All,

Thanks a lot to this community. I was reading this post

https://community.hortonworks.com/articles/83610/nifi-rest-api-flowfile-count-monitoring.html?childT...

In this post he is monitoring a single queue. So lets suppose if I have 100 queues in my NIFI do I have to create 100 monitoring process queue.

Is there any efficient way?

Thanks

Dhiren

1 ACCEPTED SOLUTION

avatar
Master Guru

In the article you mention, the ID of the connection is specified as an attribute, then the InvokeHttp processor uses it to create a URL to poll for status. In your case, you'd just need the list of connection IDs, and you can create a flow file for each connection. You can use the "/process-groups/root/connections" REST endpoint to get a list of connections at the root process group. If you want to get connections for child process groups, you'd first need to get the child process groups' IDs and use those in the connections endpoint to get the connections for each process group. That can recurse down through each child process group and may be unwieldy. For this example I'm going to assume you simply want to monitor your queues in the root process group.

With InvokeHttp and the aforementioned REST endpoint to get the connections, you'll get a JSON object back with a field called "connections", which is an array of connections. Then you can use SplitJson with a JSONPath of "$.connections", and it will create a flow file for each of the connections. Then you can use EvaluateJsonPath to extract the connection's URL using a JSONPath of "$.uri". Then you can continue using the flow described in the other article, and for each of the flow files, it will retrieve the status for that connection.

View solution in original post

3 REPLIES 3

avatar
Master Guru

In the article you mention, the ID of the connection is specified as an attribute, then the InvokeHttp processor uses it to create a URL to poll for status. In your case, you'd just need the list of connection IDs, and you can create a flow file for each connection. You can use the "/process-groups/root/connections" REST endpoint to get a list of connections at the root process group. If you want to get connections for child process groups, you'd first need to get the child process groups' IDs and use those in the connections endpoint to get the connections for each process group. That can recurse down through each child process group and may be unwieldy. For this example I'm going to assume you simply want to monitor your queues in the root process group.

With InvokeHttp and the aforementioned REST endpoint to get the connections, you'll get a JSON object back with a field called "connections", which is an array of connections. Then you can use SplitJson with a JSONPath of "$.connections", and it will create a flow file for each of the connections. Then you can use EvaluateJsonPath to extract the connection's URL using a JSONPath of "$.uri". Then you can continue using the flow described in the other article, and for each of the flow files, it will retrieve the status for that connection.

avatar
New Contributor

This is a good approach if I have handful of queues to be monitored. I have a usecase where there are many flows and in each flow I need to monitor hundreds of queues, the number of flows are increasing day by day. I want to know if there is any feasible approach for my case. I was more like thinking about having a processor that acts as a notifier.

avatar
Contributor

Hi, Ragi! Did you get the approach that you described earlier? Could you share with it?Thanks!