Unable to add new node to NiFi-1.1.0 cluster

[#1]

Hi,

I am currently using NiFi version NiFi-1.1.0 and I have a cluster with three nodes. I tried adding a fourth node to this cluster but I started getting the error of "Failed to connect node to cluster because local flow is different than cluster flow". I configured the node and made sure that it did not have a flow.xml.gz because I was trying to get it to inherit the flow.xml.gz from the cluster. When it threw the error I tried giving it the same flow.xml.gz file as the other nodes in the cluster but that also failed. I am also using a authorized-users.xml file which I call for the other nodes in their authorizers.xml file. I thought that maybe this was causing the issue so I deleted the authorized-users.xml file from the new node and tried to connect it with no flow.xml.gz file and no authorized-users.xml file, but this also failed. Would I have to stop all nodes and then start them all at the same time in order to add the new node to the cluster? Is there a way to add the new node to the cluster without getting these issues? Any suggestions would be greatly appreciated. Below I leave the error message that I keep receiving from the new node.

org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
	at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:894) ~[nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493) ~[nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) [nifi-jetty-1.1.0.jar:1.1.0]
	at org.apache.nifi.NiFi.<init>(NiFi.java:156) [nifi-runtime-1.1.0.jar:1.1.0]
	at org.apache.nifi.NiFi.main(NiFi.java:262) [nifi-runtime-1.1.0.jar:1.1.0]
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations
	at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:253) ~[nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1461) ~[nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) ~[nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:678) ~[nifi-framework-core-1.1.0.jar:1.1.0]
	at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872) ~[nifi-framework-core-1.1.0.jar:1.1.0]

[#2]

From your details I can't parse if authorizers.xml is the same on all nodes? can you verify it is?

[#3]

@Sunile Manjee, authorizers.xml is the same in all of the nodes. Since I been using the cluster since previous versions of nifi, I call the authorized-users.xml file in the legacy authorized users file path in all the nodes and all of them have the same authorized-users.xml also.

[#4]

When a new node joins the cluster it can either inherit the flow and authorizations (users + policies) from the cluster, or it has to have the exact same flow and authorizations. Lets focus on the authorizations since your error messages indicates that is where the problem was...

In order to inherit the authorizations the new node needs to have 0 users and 0 policies, this means the new node can not have a value for "Initial Admin" or "Node Identities" or "Legacy Authorized Users". Populating any of those would generate users and policies on that node. If you started up once with any of those populated then you will want to shutdown, clear out those values, and delete users.xml and authorizations.xml.

Generally once a cluster has been running for a while you have likely added users or modified policies through the UI, so the inheriting approach described above is the only way you can get a new node to join.

If you happen to be adding a new node right after you just started a new cluster, and you haven't modified any of the users or policies through the UI then you should be able to just make sure the authorizers.xml has the exact same values as the other nodes and it should generate the same users and policies and be able to join.

[#5 · Accepted solution]

@Adda Fuentes

Since you started your new node with a configured authorizers.xml file pointing at a legacy authorized-users.xml file, the users.xml and authorizations.xml files in NiFi 1.1 where generated from that rather then inheriting those files form your already running cluster.

Clear out the setting in your new nodes authorizers.xml file, remove the users.xml and authorizations.xml files, remove the flow.xml.gz file and restart this new node. It should successfully obtain these files from your existing cluster at this point.

You will need to do one additional step once this new node has joined your cluster. Since the original clusters authorizations and users will not include this new node yet, you will need to access the cluster's UI from one of the original cluster nodes using and admin account and add the new node's DN as a user and then provide that new node with all the same access policies as the your existing node have. At a minimum you need to make sure your new node is granted the "Proxy user requests" access policy:

If you do not do this the following issues could occur:

1. You will not be able to access the cluster's UI via the newly added node (you will get untrusted proxy message)

2. You will still be able to access the UI via the other nodes as long as NiFi does not switch the cluster coordinator to your newly added node. You cannot restrict NiFi from picking any node in your cluster to serve this role.

Matt