question Re: NIFI : Cluster with Multiple Instance in Archives of Support Questions (Read Only)

NIFI : Cluster with Multiple Instance

maykiwogno — Thu, 06 Oct 2016 15:15:00 GMT

Hello all,

I've trying configure two instances NIFI in the same cluster (3 nodes)

My configuration :

Instance 1 :

-bash-4.1# grep port conf/nifi.properties
nifi.remote.input.socket.port=10443
nifi.web.http.port=
nifi.web.https.port=9443
nifi.cluster.node.protocol.port=11443

-bash-4.1# more conf/zookeeper.properties

clientPort=2181

server.1=nifi001:2888:3888
server.2=nifi002:2888:3888
server.3=nifi003:2888:3888

Instance 2 :

-bash-4.1# grep port conf/nifi.properties
nifi.remote.input.socket.port=10444
nifi.web.http.port=
nifi.web.https.port=9444
nifi.cluster.node.protocol.port=11444

-bash-4.1# more conf/zookeeper.properties

clientPort=2182

server.1=nifi001:2889:3889
server.2=nifi002:2889:3889
server.3=nifi003:2889:3889

It's missing some things, because my nifi002 and nifi003 can joint the cluster.

Re: NIFI : Cluster with Multiple Instance

bbende — Thu, 06 Oct 2016 19:21:52 GMT

There could be many other things wrong, but in your ZooKeeper properties, the servers should be the same on every node. If instance #1 is pointing to 2888:3888 for all three ZK servers, then instance #2 and #3 should also be pointing to the same thing.

Re: NIFI : Cluster with Multiple Instance

maykiwogno — Thu, 06 Oct 2016 19:30:31 GMT

Thanks @Bryan Bende

Perpahs, it was not cleared, I want to start 2 instances nifi with the same cluster (node1, node2, node2)

so I would have 2 url

==> https://node1:9443 (instance1 ==> canva1)

==> https://node1:9444 (instance2 ==> canva2)

It is possible ?

Re: NIFI : Cluster with Multiple Instance

bbende — Thu, 06 Oct 2016 19:36:14 GMT

I'm not sure I understand the question..

Are you asking if you can run two NiFi clusters on the same physical hardware?

If so, what would be the reason for doing this?

Re: NIFI : Cluster with Multiple Instance

maykiwogno — Thu, 06 Oct 2016 20:18:47 GMT

I've 4 teams, i would have one canva per team.

Re: NIFI : Cluster with Multiple Instance

bbende — Thu, 06 Oct 2016 20:25:50 GMT

The way to set this up in Apache NiFi 1.0.0 is to use a single NiFi cluster with process groups for each team, and restrict the permissions appropriately so that members of a team can only work with in their given process group.

See the "Multi-Tenancy" section of this post for an example:

http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy

Re: NIFI : Cluster with Multiple Instance

rsheu — Wed, 16 Nov 2016 23:28:45 GMT

I have related question: how to deploy multiple nifi components to all the HDF cluster nodes. And, then the question to ask is just like you ask above. I use Ali's HDF ambari-bootstrap blog's approach to automatically deploy the blueprint-based HDF 2.0.1 cluster. But, it has only one NIFI in one of the specific node within the HDF cluster. I am trying to figure our how to deploy NIFI to all the HDF cluster node.