Apache Nifi- Adding new slave nodes

[#1]

Hi,

Do we need to stop all work-flows before adding a new node?

Can we add new Nifi slaves to an existing cluster? Actually I had issues adding a new Slave to an existing cluster (cluster had few workflows running).

Thanks

[#2]

@Obaid Salikeen

Nifi nodes do not talk to each other and only talk to NCM. So if you would like to add a new node, you don't need to bring the cluster down. Check the following link for details.

https://community.hortonworks.com/articles/8607/how-to-create-nifi-fault-tolerance-using-multiple.ht...

https://community.hortonworks.com/articles/8631/how-to-create-nifi-fault-tolerance-using-multiple-1....

[#3]

Only NiFi versions Apache NiFi 0.x or HDF 1.x have a NCM based cluster. NiFi versions NiFi 1.x or HDF 2.x moved to a zero master clustering which no longer relies on a NCM.

[#4 · Accepted solution]

@Obaid Salikeen

Not sure what "issues" you had when you tried to add a new node to your existing cluster.

The components (processors, connections, etc...) of an existing cluster can be running when you add new additional nodes to it. The new nodes will inherit the flow and templates from the NCM as well as the current running state of those components when it joins. But, in order for a node to successfully join a cluster the following must be true:

1. The new node either has no flow.xml.gz file and templates directory or the flow.xml.gz file and templates do not match what is currently on the NCM. (Remove flow.xml.gz file and templates dir from new node and restart node) The nifi-app.log will indicate in if a difference was found.

2. The nifi.sensitive.props.key= in the nifi.properties file must have the same value as on the NCM.

3. The NCM must be able to resolve the URL to the new node. If the nifi.web.http(s).host= was left blank on your new node, Java on that node may be reporting the hostname as localhost. Make sure valid resolvable hostnames are supplied for nifi.web.http.host=, nifi.cluster.node.address=, and nifi.cluster.node.unicast.manager.address=.

4. Both NCM and Node security protocol must match. nifi.cluster.protocol.is.secure= in nifi.properties file.

5. Firewalls must be open between NCM and Node on both HTTP(s) port and node and NCM ports.

6. New node must have all the same available java classes. If custom processors exist in your flow make sure the new node also has those custom nar/jar files included in its lib dir.

Thanks,

Matt

[#5]

Based on my experiences, I stop all data flows, get a clean (unstarted version of nifi), and copy the flow.xml.gz file from NCM into that new instance. Other actions I do include:

• delete the users.xml and authorizations.xml files on each existing node
• add the new node as an xml entry in the authorizers.xml file for all nodes including the newly added one. (all nodes should have the same authorizers.xml and zookeeper.properties
• create certificate for that node when using HTTPS mode
• add custom processors (nar files) to the lib folder if need be

[#6]

@Michael Silas

It is likely in an existing cluster that you have establish a number of user policies beyond the default "Initial Admin Identity". You do not want to delete the users.xml or authorizations.xml file at this time as you will lose all those new users and authorizations.

Instead, add the new node as a /proxy user before actually adding the node to the cluster.

You can copy the users.xml, authorizations.xml, and flow.xml.gz to you new node if you want at that time.

- Agree - create a new cert for that node. If using the NiFi CA, you can simply click teh biox for regenerate certificates in Ambari (Available in HDF 2.x releases)

- Agree that you need to be mindful of any custom nars as well as any referenced local files as these all need to copied to your new node as well.

Matt

[#7]

@Matt Clarke - you correct about adding the node that way. I play it safe by stopping all processors for my data flows, stopping all of the nodes in my cluster, adding the new node as an entry to the authorizers.xml file, copying the flow.xml.gz file to the conf directory of the new node, and deleting the users.xml and authorizations.xml files (You can do this if and iff you specify an Initial Admin in the authorizers.xml file). Of course generating a certificate for the new node and adding custom processors to the lib directory will be needed. the setback with doing it this way is that I have to re set all of the policies. A member on my team is currently writing a script that will generate authorizations.xml, users.xml and authorizers.xml files on the fly when we add a new node to save time in the long run.

Best,

Michael

[#8]

@Michael Silas

There is nothing specific in the users.xml or authorizations.xml file that is specific to any node. In Fact these files are checked on startup to make sure they are identical between all your NNiFi cluster nodes. In order for a node to successfully join a cluster the flow.xml.gz, users.xml, and authorizations.xml files must match. If you configure a new node to join an existing cluster and you have none of the above three files and have not configured the authorizers.xml file on the new node, the new node will inherit/download these three files from the cluster automatically.

Even if you have not added the new node to the "proxy user requests" global access policy yet, you should be able to still connect to the UI of your other nodes and add it afterwards.

Again, you are adding more for work for yourself by deleting the users.xml and authorizations.xml files.

Thanks,

Matt

[#9]

@Matt Clarke - I do not want to go back and forth with you, but my experiences with NiFi clusters have been slightly different than yours. I am in charge large clusters that run that have nodes on separate hosts (32+ nodes). Based on my experiences, I have had many challenges getting all of the nodes to form a cluster with all nodes. My suggestions might be more of a safest approach rather than the conventional method of adding a new node.

[#10]

@Michael Silas Once you have a running cluster you shouldn't have to modify the authorizers.xml, authorizations.xml, and users.xml to add a new node. There are two different ways you could do it..

Approach #1

1) Generate a cert for your new node

2) Go to your existing cluster and using the UI, add a new user with the DN from the cert for the new node

3) Grant the new user the policy for "proxy requests"

4) On the new node, leave the initial admin and all node identities blank, then start this node and it will since it will have 0 users and 0 policies and no flow, it will inherit everything from the cluster.

Approach #2

1) Generate a cert for your new node

2) On the new node make the authorizers.xml exactly the same as the existing cluster... meaning if you had a 3 node cluster and you are adding the fourth node, put only the 3 existing nodes as the identity and the same initial admin, this way it generates exactly the same users and policies as the running cluster which is required for it to join.

3) At this point you should be able to start the new node and have it join the cluster

4) Go into the UI and add the user for the new node and add the user to "proxy requests" policy

This blog post describes approach #2:

https://pierrevillard.com/2016/11/30/scaling-updown-a-nifi-cluster/

Overall, in order to join the cluster a new node needs one of the following conditions:

- The exact same users, groups, policies, and flow as the cluster

- No users, no groups, no policies, and no flow, in which case it will inherit everything from the cluster