Hello everyone,
I'm seeking assistance from those experienced in NiFi for a specific challenge. I use NiFi for internal tools and simple integrations within my company, organizing different functionalities into separate ProcessGroups. These functionalities include converting data types (e.g., from JSON to CSV), generating dummy data, and sending data to a partner through an API.
To make all ProcessGroups accessible through an endpoint, I've implemented a setup with a "HandleHttpRequest" processor listening on a specific port in the main canvas. Subsequently, a "RouteOnAttribute" processor is used to route requests to the appropriate ProcessGroup based on the URL path.
However, this approach has its drawbacks. The canvas becomes cluttered, requiring encapsulation of flows within more general ProcessGroups. Additionally, every time I add a new flow, I have to stop the "RouteOnAttribute," update it, link the new relationship, and then restart it (I know I can use NiFi API to accelerate this process).
In an ideal scenario, I'd like to create independent ProcessGroups, acting as "Path listeners," each capable of receiving requests without the need for linking. Essentially, I want a more dynamic and flexible setup.
I prefer to use NiFi synchronously and considered a PUB/SUB solution, but I'm uncertain about the potential delay it might introduce to the data flow.
Does anyone know of an approach I can follow, even if it involves creating a custom processor? I appreciate any insights or suggestions. Apologies if this has been addressed elsewhere; I couldn't find a solution that precisely fits my needs.
Thank you!
