unable to set up nifi cluster using NIFI 1.0.0

[#1]

Hi, @Bryan Bende i am following below post to set nifi cluster -

http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy

nifi-error.png

I am trying to set 3 node cluster using nifi 1.0.0. i have made the configuration changes as per the above post. but when i start all nodes in first go itself it threw "org.apache.nifi.controller.Uninheritable Flow Exception: Failed to connect node to cluster because local flow is different than cluster flow."

To solve this i deleted the flow.xml.gz file from all the nodes but it didn't solve the issue. Please check the attached screenshot of error.

i have encountered this issue in NIFI .0.7.0 clustering as well. it came when one of the node was not in sync with NCM. so at that time deleted flow.xml.gz file from child node only. and later after restarting it replicated the NCM's flow.xml.gz.

But in NIFI 1.0.0 as i'm not aware which node has elected as coordinator node so i had to delete flow.xml.gz file from all the nodes.

[#2]

So you deleted the flow.xml.gz from all three nodes and then tried starting again and it still wouldn't start?

[#3]

yes.. i tried restarting multiple times, it throws same exception every time.

One thing i notice is that it generate new flow.xml .gz file on all nodes but file size always differs. on one node its 0 B while on other two its of 262 B. when I extract this it has only flow.xml file that contain below structure -

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<flowController encoding-version="1.0">

<maxTimerDrivenThreadCount>10</maxTimerDrivenThreadCount> <maxEventDrivenThreadCount>5</maxEventDrivenThreadCount>

<rootGroup>

<id>4d43098d-0157-1000-27c1-1f86dcd5acf0</id>

<name>NiFi Flow</name>

<position x="0.0" y="0.0"/>

<comment/>

</rootGroup>

<controllerServices/>

<reportingTasks/>

</flowController>

[#4]

Are you using the 1.0.0 BETA or the official 1.0.0 release? My blog post was written when the BETA came out, but the official 1.0.0 release is out now which fixed problems found in the BETA. https://nifi.apache.org/download.html

[#5]

I am using the official 1.0.0 release.

[#6]

Ok, I have only seen this problem when the flows are actually different, and then deleting the different flow.xml.gz and restarting that node corrected the problem.

One thing you could try is to stop everything, delete the flow.xml.gz on each node again, then start only the first node and let it fully start and verify you can get to the UI for that node, then start the other two and see what happens.

[#7]

I just tried this approach - first node went up but other nodes are throwing same error. PFA screenshot attached.

[#8]

Can you also try to get the logs from hsuswstgdn01 ? It looks like this is a different node that was the coordinator and was logging that error that it received from hsuswstgdn01. I'm hoping that one hsuswstgdn01 there is a more detailed stacktrace for the uninheritable flow ERROR.

If you are able to, could you post all three log files instead of screenshots?

[#9]

here it is all log files. PFA . I can see 1/3 node on UI. although icons are not accessible because other two nodes are down nodes.

i have also notice this exception in logs "org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations" hence also attaching authorizers.xml file from all three node.nifi-cluster-logs.zip

[#10]

Ok that is what I was looking for, so when joining the cluster it checks the nodes flow.xml.gz and also the contents of users.xml and authorizations.xml against what the cluster has. Double check the two nodes that are not starting and see what they have in users.xml and authorizations.xml compared to the node that is starting. If they are different, you could copy over from the first node to get them in sync.

[#11]

HeyBryan, I tried the way you mentioned, still getting same error/exception. one thing i didn't understand is that users.xml and authozations.xml files are auto generated files. although i tried coping these two files from my working node to other non-working nodes as you suggested however when i let them self generate authorization.xml remain same across all nodes but inside users.xml value of identity tag differs and always set to respective node host names. please check the screenshot. do not know if this is correct behavior.nifi-users-xml.png

[#12]

users.xml and authorizations.xml are auto generated based on what you put in authorizers.xml, and authorizers.xml should be the same on all three nodes, authorizers.xml needs the Initial Admin and a Node Identity entry for all three nodes of your cluster.

In your screenshot it looks you haven't setup the Node Identities correctly, because two of them have dn01 and one of them have dn02. If theres three nodes they should have dn01, dn02, and dn03.

An example of authorizers.xml for your case would be:

<authorizer>
        <identifier>file-provider</identifier>
        <class>org.apache.nifi.authorization.FileAuthorizer</class>
        <property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Users File">./conf/users.xml</property>
        <property name="Initial Admin Identity">your admin user</property>
        <property name="Legacy Authorized Users File"></property>
        
        <property name="Node Identity 1">DN of node1</property>
        <property name="Node Identity 2">DN of node2</property>
	<property name="Node Identity 2">DN of node3</property>        
    </authorizer>

and that should be the same on all three nodes.

[#13]

Thanks for this suggestion. i want to tell you that cluster is up now. I rechecked everything. configuration was fine everywhere it just that i was setting this cluster on Azure and default nifi port 9443 was only open on one of the node hence other nodes were not coming up. I have setup a 3 node cluster now and able to see the UI on all three nodes.

Now I am going as per your post and trying to set Access Policies. As per your post " we can create a policy for the root process group by clicking the key icon in the operate palette on the left" but this key icon does not work. its not grey but it doesn't open any popup. is there any catch behind this? i checked user logs as well but find nothing.

[#14]

Clicking the lock icon should bring up the policies window:

What browser are you using? You may want to look for Javascript errors to see if something is going wrong client side?

[#15]

I'm using IE. surprisingly the certificates i generated using tls toolkit does not working in other browser.

For key icon issue i checked in browser console there comes a javascript error when i click on key icon. attached is screenshot of same. nifi-ui-error.png please check if you can help.

[#16 · Accepted solution]

IE is actually not a supported browser, Microsoft Edge is though.

See here for the list of supported browsers:

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#browser-support

[#17]

Thanks for letting me this know. But i am not able to view Nifi UI in any other browser. i have generated certs using tls toolkit as mentioned in your post and used following command -

./tls-toolkit.sh standalone -n 'host1,host2' -C 'CN=admin, OU=my cert' -o '/nifi-1.0.0/cert-target-dir'

later imported the .p12 file in browser.

except IE everywhere i'm getting secure connection failed response. please check the attached screen shot of IE and firefox nifi-security-error.png

[#18]

I think you might be running into this: https://support.mozilla.org/en-US/questions/1058856

You could try going into about:config of Firefox and adding the address into the insecure fallback hosts as that article mentions, or you could also try setting security.tls.version.min to 1.2 to see if it forces using 1.2.

I believe NiFi allows any of the TLS versions to be used, and I think when your Firefox negotiates with NiFi it ends up choosing TLSv1, and then Firefox says that is not supported anymore for some reason, at least this is what I am guessing based on the above link.

You could also just try Chrome, usually Chrome will prompt you with a warning about being unable to validate the site, which is normal because you are using a self-signed cert, and then you just add exception and continue.

For what its worth, Chrome 52 and Firefox 48.0.2 both work for me.

[#19]

Hi @Bryan Bende

I've a question regarding the issue "org.apache.nifi.controller.Uninheritable Flow Exception: Failed to connect node to cluster because local flow is different than cluster flow"

I had 3 node cluster, I was working on one of the node and other two were down. I created some data flows and now i want to replicate this to other two nodes, So I simply restarted other two assuming they should replicate the flow automatically. but I got above error. so I deleted the old flow.gz.xml, user.xml and authorization.xml from two down nodes, cleaned the log folder and restarted again. Still got the same error in logs. Attached are the logsnifi-app.txt

Am I doing something wrong here?

What is the best way to replicate the data when adding a new node in running cluster or making a down node up from existing cluster.

[#20]

What you described is the correct approach, you need to delete flow.xml.gz, users.xml, and authorizations.xml, you also need to make sure authorizers.xml is the same on all nodes. You can see from your log that it was authorizations that were the problem "Proposed Authorizations do not match current Authorizations".

You can either copy authorizers.xml from the good node to the others to ensure it is the same, or you can make sure the other nodes have no initial admin and no legacy authorized users file so that they will inherit the authorizations/users from the cluster.

[#21]

Hi @Bryan Bende

I have 3 nodes and each have different CA certificates. They are running securely. Now I am trying to cluster them.

So "Initial Admin Identity" property value of each node will remain different in "authorizers.xml" of each node and I will

make "Node Identity" for each node. Is this correct?