Support Questions

Find answers, ask questions, and share your expertise

[RESOLVED] - NIFI : Cluster NIFI 1.0.0 - add nifi node

avatar
Rising Star

Hello,

I'm trying to add new nifi node to my cluster. I have fresh install on this new node so not has flow.xml.gz not templates.

zookeeper.properties, nifi.properties are same on every nodes.

When I'm trying started new nifi node, it seems that the node is correctly connect to zookeeper server but it has an issue with flow file or authorization.

2016-11-15 14:05:01,809 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE.  Building now...
2016-11-15 14:05:02,026 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded 96 properties from system properties and environment variables
2016-11-15 14:05:02,027 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded a total of 96 properties.  Including precedence overrides effective accessible registry key size is 96
2016-11-15 14:05:02,144 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 0 records
2016-11-15 14:05:02,150 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0
2016-11-15 14:05:02,158 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 21913605242 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size
2016-11-15 14:05:02,159 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false
2016-11-15 14:05:02,324 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@41b2123e finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates
2016-11-15 14:05:02,325 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 6 milliseconds
2016-11-15 14:05:02,361 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@41b2123e checkpointed with 0 Records and 0 Swap Files in 35 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 5 millis), max Transaction ID -1
2016-11-15 14:05:02,458 INFO [main] o.a.n.c.s.server.ZooKeeperStateServer Starting Embedded ZooKeeper Peer
2016-11-15 14:05:02,680 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected...
2016-11-15 14:05:02,855 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
2016-11-15 14:05:02,973 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-15 14:05:03,034 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:host.name=nifi002
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.version=1.8.0_111
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.vendor=Oracle Corporation
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64/jre
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.class.path=/var/opt/hosting/nifi/./conf:/var/opt/hosting/nifi/./lib/commons-lang3-3.4.jar:/var/opt/hosting/nifi/./lib/logback-core-1.1.3.jar:/var/opt/hosting/nifi/./lib/log4j-over-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/nifi-properties-loader-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-runtime-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-api-1.0.0.jar:/var/opt/hosting/nifi/./lib/jcl-over-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/logback-classic-1.1.3.jar:/var/opt/hosting/nifi/./lib/jul-to-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/bcprov-jdk15on-1.54.jar:/var/opt/hosting/nifi/./lib/nifi-documentation-1.0.0.jar:/var/opt/hosting/nifi/./lib/slf4j-api-1.7.12.jar:/var/opt/hosting/nifi/./lib/nifi-framework-api-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-nar-utils-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-properties-1.0.0.jar
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.io.tmpdir=/tmp
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.compiler=<NA>
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.name=Linux
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.arch=amd64
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.version=2.6.32-642.4.2.el6.x86_64
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.name=root
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.home=/root
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.dir=/var/opt/hosting/nifi
2016-11-15 14:05:03,044 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir ./state/zookeeper/version-2 snapdir ./state/zookeeper/version-2
2016-11-15 14:05:03,051 INFO [main] o.apache.nifi.controller.FlowController The Election for Cluster Coordinator has already begun (Leader is nifi011:11443). Will not register to be elected for this role until after connecting to the cluster and inheriting the cluster's flow.
2016-11-15 14:05:03,052 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election.
2016-11-15 14:05:03,053 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
2016-11-15 14:05:03,060 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election.
2016-11-15 14:05:03,061 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started
2016-11-15 14:05:03,061 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started
2016-11-15 14:05:03,069 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-15 14:05:03,073 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.z.server.persistence.FileTxnSnapLog Snapshotting: 0x800000028 to ./state/zookeeper/version-2/snapshot.800000028
2016-11-15 14:05:03,109 INFO [SyncThread:2] o.a.z.server.persistence.FileTxnLog Creating new log file: log.800000029
2016-11-15 14:05:03,126 WARN [main] o.eclipse.jetty.util.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlets.GzipFilter
2016-11-15 14:05:03,126 WARN [main] org.eclipse.jetty.servlets.GzipFilter GzipFilter is deprecated. Use GzipHandler
2016-11-15 14:05:03,129 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@166a5659{/nifi-api,file:///var/opt/hosting/nifi/work/jetty/nifi-web-api-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.0.0.war}
2016-11-15 14:05:03,822 INFO [main] /nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath
2016-11-15 14:05:03,846 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@65cc3902{/nifi-content-viewer,file:///var/opt/hosting/nifi/work/jetty/nifi-web-content-viewer-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.0.0.war}
2016-11-15 14:05:03,847 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@a08e41b{/nifi-docs,null,AVAILABLE}
2016-11-15 14:05:03,904 INFO [main] /nifi-docs No Spring WebApplicationInitializer types detected on classpath
2016-11-15 14:05:03,906 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@e08d871{/nifi-docs,file:///var/opt/hosting/nifi/work/jetty/nifi-web-docs-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.0.0.war}
2016-11-15 14:05:03,961 INFO [main] / No Spring WebApplicationInitializer types detected on classpath
2016-11-15 14:05:03,963 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@7962c1d5{/,file:///var/opt/hosting/nifi/work/jetty/nifi-web-error-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.0.0.war}
2016-11-15 14:05:03,968 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@377cbdae(nifi-key,h=[nifi002],w=[]) for SslContextFactory@37a939bc(file:///var/opt/hosting/nifi/conf/keystore.jks,file:///var/opt/hosting/nifi/conf/truststore.jks)
2016-11-15 14:05:03,981 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@37aace65{SSL,[ssl, http/1.1]}{nifi002:9443}
2016-11-15 14:05:03,982 INFO [main] org.eclipse.jetty.server.Server Started @43215ms
2016-11-15 14:05:05,035 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...
2016-11-15 14:05:05,049 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 11443
2016-11-15 14:05:05,160 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: nifi002:9443
2016-11-15 14:05:05,165 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi011:11443; will use this address for sending heartbeat messages
2016-11-15 14:05:05,563 INFO [Process Cluster Protocol Request-1] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi002:9443 changed from null to NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28]
2016-11-15 14:05:05,568 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 99d259cd-736b-46da-9c8a-20419f1a31f8 (type=NODE_STATUS_CHANGE, length=1182 bytes) from nifi011 in 149 millis
2016-11-15 14:05:05,653 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {nifi002:9443=NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28]} to {nifi012:9443=NodeConnectionStatus[nodeId=nifi012:9443, state=CONNECTED, updateId=17], nifi013:9443=NodeConnectionStatus[nodeId=nifi013:9443, state=CONNECTED, updateId=18], nifi014:9443=NodeConnectionStatus[nodeId=nifi014:9443, state=CONNECTED, updateId=19], nifi001:9443=NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow., updateId=26], nifi011:9443=NodeConnectionStatus[nodeId=nifi011:9443, state=CONNECTED, updateId=20], nifi002:9443=NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28]}
2016-11-15 14:05:05,662 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
        at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879) ~[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493) ~[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) [nifi-jetty-1.0.0.jar:1.0.0]
        at org.apache.nifi.NiFi.<init>(NiFi.java:152) [nifi-runtime-1.0.0.jar:1.0.0]
        at org.apache.nifi.NiFi.main(NiFi.java:243) [nifi-runtime-1.0.0.jar:1.0.0]
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations
        at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252) ~[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435) ~[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) ~[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671) ~[nifi-framework-core-1.0.0.jar:1.0.0]
        at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857) ~[nifi-framework-core-1.0.0.jar:1.0.0]
        ... 4 common frames omitted
2016-11-15 14:05:05,663 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator nifi002:9443 requested disconnection from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
2016-11-15 14:05:05,664 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi002:9443 changed from NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28] to NodeConnectionStatus[nodeId=nifi002:9443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow., updateId=28]
2016-11-15 14:05:05,803 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for nifi002:9443 -- Node disconnected from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
2016-11-15 14:05:05,816 ERROR [Shutdown Cluster Coordinator] o.a.z.server.NIOServerCnxnFactory Thread Thread[Shutdown Cluster Coordinator,5,main] died
java.lang.NullPointerException: null

thanks for helps

1 ACCEPTED SOLUTION

avatar
Master Mentor
@mayki wogno

The reported error message is a little misleading, but the key to your issue is reported in the "Caused by" portion of the stack trace:

Causedby: org.apache.nifi.controller.UninheritableFlowException:ProposedAuthorizerisnot inheritable by the flow controller because of Authorizer differences:ProposedAuthorizationsdonot match current Authorizations

Verify that the authorizers.xml file on your new node matches the configuration on your other existing connected nodes.

Remove the users.xml and authorizations.xml files from your new node as well be restarting. The new Node should be getting these files from the cluster when it joins. If you continue to see the same error message on restart, try copying the users.xml and authorizations.xml files from one of your existing connected nodes to this new node and restart again.

Thanks,

Matt

View solution in original post

8 REPLIES 8

avatar
Super Collaborator

see if this works for you.

avatar
Master Mentor

@Karthik Narayanan

The link you reference is good for troubleshooting adding new nodes in NiFi 0.x and HDF 1.x versions of NiFi which had an NCM. NiFi 1.x and HDF 2.x version are very different now and do not have an NCM.

avatar
Rising Star

@Karthik Narayanan thanks for the url.

They said about templates, what are the templates in the nifi directories ?

I've removed only conf/flow.xml.gz

avatar
Super Collaborator

@mayki wogno the thread also mentions deleting a flow.tar file, did you try that.

avatar
Master Mentor
@mayki wogno

The reported error message is a little misleading, but the key to your issue is reported in the "Caused by" portion of the stack trace:

Causedby: org.apache.nifi.controller.UninheritableFlowException:ProposedAuthorizerisnot inheritable by the flow controller because of Authorizer differences:ProposedAuthorizationsdonot match current Authorizations

Verify that the authorizers.xml file on your new node matches the configuration on your other existing connected nodes.

Remove the users.xml and authorizations.xml files from your new node as well be restarting. The new Node should be getting these files from the cluster when it joins. If you continue to see the same error message on restart, try copying the users.xml and authorizations.xml files from one of your existing connected nodes to this new node and restart again.

Thanks,

Matt

avatar
Rising Star

@Karthik Narayanan : except flow.xml.gz, i don't have file flow.tar

avatar
Rising Star

It worked to copy users and authorization.

thanks all.

avatar
Super Collaborator

dont do the SCP part. the files will get downloaded from the NCM when the node connects to it.