HDF 3.0 - Issue with Adding a new NiFi Node(s) to an existing NiFi cluster via Ambari

[#1]

Hello,

While trying to add nodes to my existing NiFi cluster with Ambari, I am getting below error in 3.0 version of HDF. It used to work fine in HDF-2.x version. The issue is related to flow differences, but error messages indicates its the issue with AmbariReportingTaskAmbariReport finger print. Ambari used to take care of this in HDF-2.x version.

Am I missing something here? Any additional steps which I should be doing in 3.0? I cross checked it with below article for HDF-2.0 by @Matt Clarke, except my cluster is unsecured no other differences:

https://community.hortonworks.com/articles/80284/hdf-2x-adding-a-new-nifi-node-to-an-existing-secur....

Any pointers will be appreciated!

2017-07-19 12:39:03,469 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:936) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800) at org.apache.nifi.NiFi.<init>(NiFi.java:160) at org.apache.nifi.NiFi.main(NiFi.java:267) Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Flows: Local Fingerprint: 7c84501d-d10c-407c-b9f3-1d80e38fe36a3b80ba0f-a6c0-48db-b721-4dbc04cef28eorg.apache.nifi.reporting.ambari.AmbariReportingTaskAmbariReport Cluster Fingerprint: 7c84501d-d10c-407c-b9f3-1d80e38fe36a428574ae-015d-1000-0000-0000454185d7org.apache.nifi.processors.standard.TailFileNO_VALUEorg.apache.n at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:259) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1576) at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722) at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:911) ... 4 common frames omitted

Thanks

[#2]

@Jobin George

On your new node, do you have flow.xml.gz? If yes, can you delete it and try adding the node again.

[#3]

hello @mqureshi,

It was a fresh node. there were no flow.xml.gz file present at first, when I started the instance it created a 480b flow.xml.gz file later grow to 593b. I tried deleting it and start/restart NiFi on the fresh node, again a 480b file was created instead of 11880b cluster flow.xml.gz (copying that 11880b cluster flow should have been done by coordinator node but looks like its been done by Ambari - I guess)

node4 is the new node and node1 is existing node.

[#4]

@Jobin George

Can you shutdown the cluster, then delete your flow file from node 4, then add the node in Ambari, before starting verify new flow file is not there and then start the cluster. I know this is wrong because we should be able to add a node without bringing the cluster down but I just want to see what might make it work.

[#5 · Accepted solution]

@Jobin George

You will have to copy over the current flow.xml.gz from node1 to node4, then start NiFi on node4.

[#6]

@Wynner , Is this a mandatory manual step for NiFi node addition while done via Ambari? I am sure this worked fine with out copying flow.xml.gz in HDF-2.x. Or some work is going on to fix it?

[#7]

@Jobin George

That is a step I have always had to do when adding a new node to an existing cluster when using Ambari

[#8]

@Jobin George

The fix is being worked currently so you don't have to copy over the flow.xml.gz manually.

[#9]

Good to know that!!.. thanks for the update @Wynner

[#10]

@Jobin George

The issue here is caused by the following....

1. Ambari metrics have been enabled.

2. On start of a NiFi Node, If Ambari detects a flow.xml.gz fiel does not exist, it creates a flow.xml.gz that contains only the AmbariReportingTask to support the enabling of Ambari metrics from this NiFi.

3. Then NiFi is started and NiFi's normal startup procedure occurs. During that process NiFi detects the flow.xml.gz on this new node does not match the flow.xml.gz on the cluster. Node will shut back down.

Aside from just manually copying the flow.xml.gz from an existing cluster node, another workaround is to make sure the flow.xml.gz fiel is not there and start the new node via NiFi's command line start manually to bypass the Ambari flow.xml.gz file generation.

Thanks,

Matt