NIFI Architectural Approach - Independent Flows

[#1]

I am looking at creating several logical (independent) NIFI flows. Because of their independence I could imagine executing each flow within a separate instance of NIFI. Of course, I can also create these flows within a single instance of NIFI.

I'm trying to understand the best way to organize my project. Would you suggest that I run multiple instances of NIFI on my platform (one for each independent flow)? Or should I run a single instance of NIFI, and have it manage each independent flow?

Part of what I am struggling with is the graphical management of the flows within a single instance of NIFI. The graphical interface works well for a small number of processors, it starts to feel cumbersome as the number of processors and flows increases. (I'm used to the neat structure of a software program).

Ultimately the flows I'm creating would be part of a mission critical system. I want to make sure I adopt the best practice in the formulation of the flows and the management environment.

[#2 · Accepted solution]

@David Sargrad

-

Would potentially be a waste of resources to use separate NiFi instances fro each of your dataflows. A single instance also provide no HA at all.

A better approach is to setup a NiFi cluster where you run multiple dataflows.

To help keep your dataflows organized, user typically make use of a "Process group" for each unique dataflow.

On the root canvas you simply have a Process group for each unique dataflow you create to keep the UI clean and manageable.

This type of setup would also allow you to easily "version control" each of these process groups independently to a NiFi registry.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

[#3]

I like your suggestion. Ty.

[#4]

@Matt Clarke Hi Matt. One question I'd like to get your perspective on.. Assuming that I manage independent flows within a single NIFI cluster, is it your experience that I can use the NIFI registry to properly version and manage the independent flows that are processed within that cluster? The inability to manage each independent flow as a versioned flow, could potentially drive me to using multiple NIFI clusters (and to assign each cluster one or several flows). I'm concerned about the overall complexity of the processing that is assigned to a single cluster.