unable to set up nifi cluster using NIFI 1.0.0

[#1]

Hi, @Bryan Bende i am following below post to set nifi cluster -

http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy

nifi-error.png

I am trying to set 3 node cluster using nifi 1.0.0. i have made the configuration changes as per the above post. but when i start all nodes in first go itself it threw "org.apache.nifi.controller.Uninheritable Flow Exception: Failed to connect node to cluster because local flow is different than cluster flow."

To solve this i deleted the flow.xml.gz file from all the nodes but it didn't solve the issue. Please check the attached screenshot of error.

i have encountered this issue in NIFI .0.7.0 clustering as well. it came when one of the node was not in sync with NCM. so at that time deleted flow.xml.gz file from child node only. and later after restarting it replicated the NCM's flow.xml.gz.

But in NIFI 1.0.0 as i'm not aware which node has elected as coordinator node so i had to delete flow.xml.gz file from all the nodes.

[#2]

So you deleted the flow.xml.gz from all three nodes and then tried starting again and it still wouldn't start?

[#3]

yes.. i tried restarting multiple times, it throws same exception every time.

One thing i notice is that it generate new flow.xml .gz file on all nodes but file size always differs. on one node its 0 B while on other two its of 262 B. when I extract this it has only flow.xml file that contain below structure -

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<flowController encoding-version="1.0">

<maxTimerDrivenThreadCount>10</maxTimerDrivenThreadCount> <maxEventDrivenThreadCount>5</maxEventDrivenThreadCount>

<rootGroup>

<id>4d43098d-0157-1000-27c1-1f86dcd5acf0</id>

<name>NiFi Flow</name>

<position x="0.0" y="0.0"/>

<comment/>

</rootGroup>

<controllerServices/>

<reportingTasks/>

</flowController>

[#4]

Are you using the 1.0.0 BETA or the official 1.0.0 release? My blog post was written when the BETA came out, but the official 1.0.0 release is out now which fixed problems found in the BETA. https://nifi.apache.org/download.html

[#5]

I am using the official 1.0.0 release.

[#6]

Ok, I have only seen this problem when the flows are actually different, and then deleting the different flow.xml.gz and restarting that node corrected the problem.

One thing you could try is to stop everything, delete the flow.xml.gz on each node again, then start only the first node and let it fully start and verify you can get to the UI for that node, then start the other two and see what happens.

[#7]

I just tried this approach - first node went up but other nodes are throwing same error. PFA screenshot attached.

[#8]

Can you also try to get the logs from hsuswstgdn01 ? It looks like this is a different node that was the coordinator and was logging that error that it received from hsuswstgdn01. I'm hoping that one hsuswstgdn01 there is a more detailed stacktrace for the uninheritable flow ERROR.

If you are able to, could you post all three log files instead of screenshots?

[#9]

here it is all log files. PFA . I can see 1/3 node on UI. although icons are not accessible because other two nodes are down nodes.

i have also notice this exception in logs "org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations" hence also attaching authorizers.xml file from all three node.nifi-cluster-logs.zip

[#10]

Ok that is what I was looking for, so when joining the cluster it checks the nodes flow.xml.gz and also the contents of users.xml and authorizations.xml against what the cluster has. Double check the two nodes that are not starting and see what they have in users.xml and authorizations.xml compared to the node that is starting. If they are different, you could copy over from the first node to get them in sync.