What are best practices for NiFi development in a team context?

[#1]

I would like to introduce Nifi as a tool for controlling a top-level work flow, but I want it to be something that my whole team can access and maintain, and I'm wondering about best practices in this context.

For example, we currently have a single Nifi instance with a single flow on a shared server.

So anyone on the team can RDP to the server and see/edit the flow at localhost:8080 But only one person at a time.

But what if we want multiple flows and the ability for multiple devs to have access at the same time?

At a high level, it looks like we could run multiple instances of Nifi and just have a record somewhere that localhost:8080 is prod and localhost:8090 is dev or something like that.

But that still doesn't allow admin A to work on prod and admin B to work on Dev at the same time. They would have to make changes on separate machine and then deploy the XML.

Even if we opened up the ports so that Nifi is accessible through the browser on a remote machine, how does it work if 2 devs are editing at the same time? Is that ok as long as they are in separate process groups?

I'm trying to understand the options and best practices for this scenario.

Thanks!

[#2 · Accepted solution]

@Zack Riesland

Multiple users can access and make changes on the canvas at the same time. The only time that acton is blocked is if both users are trying to edit the exact same component at the same time. First user in such a scenario to hit "accept" on their changes wins. The other users change will be lost and they will be force to refresh and try again.
-

When NiFi is secured, you have the ability to setup very granular access controls. You can restrict different authenticated users for example to only have access to specific process groups. This would prevent users from editing components belonging to other users.

-

Thanks,

Matt

-

If you found this answer addressed your question, please take moment to login and click "accept" below the answer.

[#3]

@Zack Riesland

***Forum Tip: Try to avoid responding to an existing "Answer" by starting a new "Answer". There is no guaranteed order to answer, so conversation may get difficult to follow. Instead use the "Add comment" below an answer to respond.

-

That works as long as you have discipline amongst your various devs. With NiFi being unsecured there will be no controls preventing one user from modifying another users flows or deleting other users components. There also will not be an audit trail, as every change is logged as being done by anonymous.

-

Otherwise, a process group does provide user with what appears to be their own work space.

-

Keep in mind that it is still a single NiFi JVM, so every dataflow (no matter whom built it) is operating unders the same shared resource constraints (CPU, HEAP, etc..).

-

Thanks,

Matt

[#4]

Thanks @Matt Clarke

My mistake on the response - I clicked "reply" but apparently managed to type in the wrong box...

I have one followup question since you seem to know Nifi -

Simply opening access to port 8080 on the Nifi server doesn't appear to be sufficient for making it accessible to other computers on the same network.

I've been looking for some instructions, and everything I've found points to setting up HTTPS, certificates, keys, etc.

(like this https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy)

Is that the only option?

For reference, this is running in a VPC and only machines with VPN access can see the server at all.

Thanks!

[#5]

@Zack Riesland

Hope I know NiFi, been working with it almost since the beginning (8+ years now) 🙂

-

The only port that needs to be open for users to access the NiFi UI of an unsecured NiFi is which ever port is defined in the nifi.web.http.port= property in the nifi.properties file.

-

That being said, is your server hosting NIFi multi-homed?

The nifi.web.http.host= property is typically configured with the FQDN for your host. When left blank NiFi will bind to every interface on your host. When set it will bind to only the interface which the FQDN is assign to. So maybe you are having issues because NiFi has bound to an interface you cannot reach from other computers on your network?

-

During startup you will see a line that says:

org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:

immediately following that line will be a list of URLs that can be used to access this running NiFi instance.

-

Some components in NiFi that create their own listening port (example: ListenHTTP) would have a unique port that would need to be opened before they could be accessed by external systems as well

-

Thanks,

Matt

[#6]

Thanks @Matt Clarke

You must be back from the NSA days 🙂

Your message is helpful, but I'm still not able to access the from my browser laptop.

Here's what I've got:

I have a RHEL 7.5 server running in EC2, in a VPC.

It's running Nifi 1.6.0 using all vanilla settings. I can access the server using NoMachine and interract with Nifi in the browser directly on the machine.

I added a SecurityGroup to open port 8080.

As you said, the logs list about 4 different URLs - they are all different IPs associated with the machine.

But none of them work from my laptop (which is in the VPC via VPN).

I also tried setting the nifi.web.http.host value, and I also tried changing to a different port (restarting after each change).

I even tried setting the Security Group to allow "all traffic" from "everywhere".

So I don't think ports are the issue.

(Interestingly, if I set the nifi.web.http.host value, I am no longer able to access nifi in the browser on the host machine using 'localhost')

So... Any other ideas? I'm feeling a little stuck...

[#7]

Thanks Matt,

My issue was firewall related.

I'm all set now. Thanks for your help!

[#8]

@Zack Riesland

Your problem is definitely external to NiFi.

-

Perhaps a NAT issue between your outside and the instance running in EC2.

Maybe Security Groups configuration issue?

Maybe Network access control list issue?

-

https://aws.amazon.com/premiumsupport/knowledge-center/instance-vpc-troubleshoot/

-

From your external machine you should be able to use command like telnet, openssl, or netcat to verify ability to connect to the NiFi URL endpoint in EC2.

-

Sorry I won't be much help with EC2 specific issues.

You could always start a new community thread asking for help specifically with accessing http endpoints within a EC2 in a VPC.

-

Thanks,

Matt

[#9]

Thanks @Matt Clarke

So I'm thinking I'll open up the port so that different devs can access the flow through the browser (it's all protected by VPN) and utilize process groups to help isolate distinct pieces.

Does that sound like a good plan?