Nifi with Multiple Data Flow

[#1]

We are currently building a multi tenant Hadoop platform, and would like to use Nifi as our data flow management tool.

I am having a hard time to figure out how can you create more than one Nifi data flow. The reason I want to do that is due to the platform potentially being used by different teams, we don't want everybody just keep adding processor groups to the same flow, and we don't want all the users to be able to see other peoples flow.

I am still using the the anonymous authentication, so I don't know if the flows will be per use based once I configured the User Authentication, which I am working on. Any help is appreciated.

[#2 · Accepted solution]

Vance,

We completely agree with you. NiFi already supports some powerful security and multi-role authorization capabilities. But as you mention we should support multiple different groups with different levels of access to various parts of the flow. That is an important roadmap item and work is underway. You can see a bit about the nifi community thinking on this wiki page https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow and there are related threads such as https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface and https://cwiki.apache.org/confluence/display/NIFI/Support+Authorizer+API

If you need help setting up secure NiFi you can read more here https://community.hortonworks.com/articles/886/securing-nifi-step-by-step.html and in the administration guide https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security-configuration

Thanks

Joe

[#3]

Thanks jwitt. I understand, but the parts are talking about supporting different levels of access within one flow for different users. What I am looking for is creating different flows for the same user or different users. Is the latter currently possible for Nifi?

Or is having one flow per instance is the underlying basic assumption of Nifi?

[#4]

Vance currently in NiFi any user with the DFM permission can create as many flows as necessary. It is not uncommon for a single instance of NiFi to be handling hundreds or more processors representing what can be dozens or hundreds of distinct dataflows. It is also quite common for people to be surprised by that however a lot of effort has gone into the design of the repositories, threading model, and user interface to allow it to support a wide variety of functions and flows. It is certainly a solid compliment to the powerful analysis and processing platforms that systems like Storm and Spark provide or the storage/access systems that Kafka and HDFS provide.