- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
NiFi queue monitoring
- Labels:
-
Apache NiFi
Created 02-08-2018 05:08 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All,
Thanks a lot to this community. I was reading this post
In this post he is monitoring a single queue. So lets suppose if I have 100 queues in my NIFI do I have to create 100 monitoring process queue.
Is there any efficient way?
Thanks
Dhiren
Created 02-08-2018 05:35 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In the article you mention, the ID of the connection is specified as an attribute, then the InvokeHttp processor uses it to create a URL to poll for status. In your case, you'd just need the list of connection IDs, and you can create a flow file for each connection. You can use the "/process-groups/root/connections" REST endpoint to get a list of connections at the root process group. If you want to get connections for child process groups, you'd first need to get the child process groups' IDs and use those in the connections endpoint to get the connections for each process group. That can recurse down through each child process group and may be unwieldy. For this example I'm going to assume you simply want to monitor your queues in the root process group.
With InvokeHttp and the aforementioned REST endpoint to get the connections, you'll get a JSON object back with a field called "connections", which is an array of connections. Then you can use SplitJson with a JSONPath of "$.connections", and it will create a flow file for each of the connections. Then you can use EvaluateJsonPath to extract the connection's URL using a JSONPath of "$.uri". Then you can continue using the flow described in the other article, and for each of the flow files, it will retrieve the status for that connection.
Created 02-08-2018 05:35 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In the article you mention, the ID of the connection is specified as an attribute, then the InvokeHttp processor uses it to create a URL to poll for status. In your case, you'd just need the list of connection IDs, and you can create a flow file for each connection. You can use the "/process-groups/root/connections" REST endpoint to get a list of connections at the root process group. If you want to get connections for child process groups, you'd first need to get the child process groups' IDs and use those in the connections endpoint to get the connections for each process group. That can recurse down through each child process group and may be unwieldy. For this example I'm going to assume you simply want to monitor your queues in the root process group.
With InvokeHttp and the aforementioned REST endpoint to get the connections, you'll get a JSON object back with a field called "connections", which is an array of connections. Then you can use SplitJson with a JSONPath of "$.connections", and it will create a flow file for each of the connections. Then you can use EvaluateJsonPath to extract the connection's URL using a JSONPath of "$.uri". Then you can continue using the flow described in the other article, and for each of the flow files, it will retrieve the status for that connection.
Created 08-21-2018 06:30 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is a good approach if I have handful of queues to be monitored. I have a usecase where there are many flows and in each flow I need to monitor hundreds of queues, the number of flows are increasing day by day. I want to know if there is any feasible approach for my case. I was more like thinking about having a processor that acts as a notifier.
Created 10-09-2018 07:56 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, Ragi! Did you get the approach that you described earlier? Could you share with it?Thanks!
