Nifi Cluster

[#1]

I have Nifi 1.9.2 and I'm trying to run it into 3 different ports on the same machine (Localhost). I can't seem to find Anything on the internet that can help me with that.

Can we do this?

I need your help guys, 

Thanks in advance

[#2]

I believe it is possible.  You will need to create 3 separate installs of NiFI on same node, each with different ports.  You will also need to do some custom configuration to make sure they are communicating together as a cluster.   This would be very advanced.  Also, putting on same node is not recommended.

I would Highly Recommend you to build a 3 node cluster using Ambari, and install NiFi in this manner as it will make things very easy.

[#3]

@ChampagneM12 

Running multiple NiFi nodes within the same NiFi cluster on the same system is not recommended, but can be done.

This is possible by editing the nifi.properties file for each NiFi node so that is binds to its own unique HTTP ports for the following settings:
nifi.remote.input.socket.port=
nifi.web.http(s).port=

nifi.cluster.node.protocol.port=

nifi.cluster.node.load.balance.port=

On startup NiFi will bind to these ports and multiple nodes on the same server can not bind to the same port.

Also keep in mind that multple NiFi instances can NOT share resources like any of the repositories (database, flowfile, content, or provenance), local state directories, etc. so make sure those are all set to unique paths per node in the nifi configuration files (nifi.properties, state-management.xml, authorizers.xml)

This will allow you to have multiple nodes loaded on the same server in same NiFi cluster. You will however potentially run in to issues when you start building your dataflows...

Each instance will run its own copy of the data flows you construct.  So any processor or controller service you add that sets up a listener will not work as only one node in your cluster will successfully bind to the configured port (there is not workaround for this). 
 
So total success here is going to in part depend on what kind of data flows you will be building.

Hope this helps,

Matt

[#4]

@MattWho @stevenmatison, Hey guys, I thank you for responding on such a short notice.

you are both not recommending this. I hear you both.

The goal of my architechture is to have three nifi as starting point, they all communicating with each other. but if one goes down (the leader), one of the two that are still running ensure the safety and digesting of data like if nothing is hapenning. 

I used the same Architecture for Kafka. In kafka we have brokers. the zookeeper elect one as a leader. and if the leader fails and is not longer available, zookeeper elect a new one to take the lead and digest Data. knowing that we dont loose any data at any point. in kafka I created three brokers one on localhost:9092, localhost:9093, localhost:9094

is this logic can not work on nifi? if no, what yall are saying is I should work with three different machines will that guarentee the safety and digesting of data? 

I thank you in advance guys!

PS: the use of both Nifi and Kafka is for a big company. the goal is to manage failovers.

[#5]

Okey @MattWho, understood!

Can you help me, if you can of course, try to configure a 2 nifi instance communication.

I have 3 Nifi Install and I want them to communicate with each others, but I'm trying to configure just two for now so I can see if it works. 

I have configured one on 8080,

# Site to Site properties
nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8080
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=localhost
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=

and one on 8081

# Site to Site properties
nifi.remote.input.host=localhost
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8081
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.contents.cache.expiration=30 secs

# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=localhost
nifi.web.http.port=8081
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=

The problem is it shuts down just after I run it. Am I doing something wrong?

[#6]

@ChampagneM12 

Please start a new thread for your new issue.  We try to keep one question to a thread to avoid confusion and help users who made also have the same embedded question you are asking find it easier. Ping me in your new thread and I would be happy to help as much as i can.

Matt

[#7]

Alright!

Thank you @MattWho