Created 11-15-2016 01:19 PM
Hello,
I'm trying to add new nifi node to my cluster. I have fresh install on this new node so not has flow.xml.gz not templates.
zookeeper.properties, nifi.properties are same on every nodes.
When I'm trying started new nifi node, it seems that the node is correctly connect to zookeeper server but it has an issue with flow file or authorization.
2016-11-15 14:05:01,809 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE. Building now... 2016-11-15 14:05:02,026 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded 96 properties from system properties and environment variables 2016-11-15 14:05:02,027 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded a total of 96 properties. Including precedence overrides effective accessible registry key size is 96 2016-11-15 14:05:02,144 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 0 records 2016-11-15 14:05:02,150 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0 2016-11-15 14:05:02,158 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 21913605242 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2016-11-15 14:05:02,159 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2016-11-15 14:05:02,324 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@41b2123e finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2016-11-15 14:05:02,325 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 6 milliseconds 2016-11-15 14:05:02,361 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@41b2123e checkpointed with 0 Records and 0 Swap Files in 35 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 5 millis), max Transaction ID -1 2016-11-15 14:05:02,458 INFO [main] o.a.n.c.s.server.ZooKeeperStateServer Starting Embedded ZooKeeper Peer 2016-11-15 14:05:02,680 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2016-11-15 14:05:02,855 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2016-11-15 14:05:02,973 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2016-11-15 14:05:03,034 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:host.name=nifi002 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.version=1.8.0_111 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.vendor=Oracle Corporation 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64/jre 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.class.path=/var/opt/hosting/nifi/./conf:/var/opt/hosting/nifi/./lib/commons-lang3-3.4.jar:/var/opt/hosting/nifi/./lib/logback-core-1.1.3.jar:/var/opt/hosting/nifi/./lib/log4j-over-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/nifi-properties-loader-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-runtime-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-api-1.0.0.jar:/var/opt/hosting/nifi/./lib/jcl-over-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/logback-classic-1.1.3.jar:/var/opt/hosting/nifi/./lib/jul-to-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/bcprov-jdk15on-1.54.jar:/var/opt/hosting/nifi/./lib/nifi-documentation-1.0.0.jar:/var/opt/hosting/nifi/./lib/slf4j-api-1.7.12.jar:/var/opt/hosting/nifi/./lib/nifi-framework-api-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-nar-utils-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-properties-1.0.0.jar 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.io.tmpdir=/tmp 2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.compiler=<NA> 2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.name=Linux 2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.arch=amd64 2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.version=2.6.32-642.4.2.el6.x86_64 2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.name=root 2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.home=/root 2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.dir=/var/opt/hosting/nifi 2016-11-15 14:05:03,044 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir ./state/zookeeper/version-2 snapdir ./state/zookeeper/version-2 2016-11-15 14:05:03,051 INFO [main] o.apache.nifi.controller.FlowController The Election for Cluster Coordinator has already begun (Leader is nifi011:11443). Will not register to be elected for this role until after connecting to the cluster and inheriting the cluster's flow. 2016-11-15 14:05:03,052 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election. 2016-11-15 14:05:03,053 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2016-11-15 14:05:03,060 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election. 2016-11-15 14:05:03,061 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2016-11-15 14:05:03,061 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2016-11-15 14:05:03,069 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED 2016-11-15 14:05:03,073 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.z.server.persistence.FileTxnSnapLog Snapshotting: 0x800000028 to ./state/zookeeper/version-2/snapshot.800000028 2016-11-15 14:05:03,109 INFO [SyncThread:2] o.a.z.server.persistence.FileTxnLog Creating new log file: log.800000029 2016-11-15 14:05:03,126 WARN [main] o.eclipse.jetty.util.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlets.GzipFilter 2016-11-15 14:05:03,126 WARN [main] org.eclipse.jetty.servlets.GzipFilter GzipFilter is deprecated. Use GzipHandler 2016-11-15 14:05:03,129 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@166a5659{/nifi-api,file:///var/opt/hosting/nifi/work/jetty/nifi-web-api-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.0.0.war} 2016-11-15 14:05:03,822 INFO [main] /nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath 2016-11-15 14:05:03,846 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@65cc3902{/nifi-content-viewer,file:///var/opt/hosting/nifi/work/jetty/nifi-web-content-viewer-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.0.0.war} 2016-11-15 14:05:03,847 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@a08e41b{/nifi-docs,null,AVAILABLE} 2016-11-15 14:05:03,904 INFO [main] /nifi-docs No Spring WebApplicationInitializer types detected on classpath 2016-11-15 14:05:03,906 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@e08d871{/nifi-docs,file:///var/opt/hosting/nifi/work/jetty/nifi-web-docs-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.0.0.war} 2016-11-15 14:05:03,961 INFO [main] / No Spring WebApplicationInitializer types detected on classpath 2016-11-15 14:05:03,963 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@7962c1d5{/,file:///var/opt/hosting/nifi/work/jetty/nifi-web-error-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.0.0.war} 2016-11-15 14:05:03,968 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@377cbdae(nifi-key,h=[nifi002],w=[]) for SslContextFactory@37a939bc(file:///var/opt/hosting/nifi/conf/keystore.jks,file:///var/opt/hosting/nifi/conf/truststore.jks) 2016-11-15 14:05:03,981 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@37aace65{SSL,[ssl, http/1.1]}{nifi002:9443} 2016-11-15 14:05:03,982 INFO [main] org.eclipse.jetty.server.Server Started @43215ms 2016-11-15 14:05:05,035 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow... 2016-11-15 14:05:05,049 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 11443 2016-11-15 14:05:05,160 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: nifi002:9443 2016-11-15 14:05:05,165 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi011:11443; will use this address for sending heartbeat messages 2016-11-15 14:05:05,563 INFO [Process Cluster Protocol Request-1] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi002:9443 changed from null to NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28] 2016-11-15 14:05:05,568 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 99d259cd-736b-46da-9c8a-20419f1a31f8 (type=NODE_STATUS_CHANGE, length=1182 bytes) from nifi011 in 149 millis 2016-11-15 14:05:05,653 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {nifi002:9443=NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28]} to {nifi012:9443=NodeConnectionStatus[nodeId=nifi012:9443, state=CONNECTED, updateId=17], nifi013:9443=NodeConnectionStatus[nodeId=nifi013:9443, state=CONNECTED, updateId=18], nifi014:9443=NodeConnectionStatus[nodeId=nifi014:9443, state=CONNECTED, updateId=19], nifi001:9443=NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow., updateId=26], nifi011:9443=NodeConnectionStatus[nodeId=nifi011:9443, state=CONNECTED, updateId=20], nifi002:9443=NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28]} 2016-11-15 14:05:05,662 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) [nifi-jetty-1.0.0.jar:1.0.0] at org.apache.nifi.NiFi.<init>(NiFi.java:152) [nifi-runtime-1.0.0.jar:1.0.0] at org.apache.nifi.NiFi.main(NiFi.java:243) [nifi-runtime-1.0.0.jar:1.0.0] Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671) ~[nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857) ~[nifi-framework-core-1.0.0.jar:1.0.0] ... 4 common frames omitted 2016-11-15 14:05:05,663 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator nifi002:9443 requested disconnection from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. 2016-11-15 14:05:05,664 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi002:9443 changed from NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28] to NodeConnectionStatus[nodeId=nifi002:9443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow., updateId=28] 2016-11-15 14:05:05,803 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for nifi002:9443 -- Node disconnected from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow. 2016-11-15 14:05:05,816 ERROR [Shutdown Cluster Coordinator] o.a.z.server.NIOServerCnxnFactory Thread Thread[Shutdown Cluster Coordinator,5,main] died java.lang.NullPointerException: null
thanks for helps
Created 11-15-2016 01:43 PM
The reported error message is a little misleading, but the key to your issue is reported in the "Caused by" portion of the stack trace:
Causedby: org.apache.nifi.controller.UninheritableFlowException:ProposedAuthorizerisnot inheritable by the flow controller because of Authorizer differences:ProposedAuthorizationsdonot match current Authorizations
Verify that the authorizers.xml file on your new node matches the configuration on your other existing connected nodes.
Remove the users.xml and authorizations.xml files from your new node as well be restarting. The new Node should be getting these files from the cluster when it joins. If you continue to see the same error message on restart, try copying the users.xml and authorizations.xml files from one of your existing connected nodes to this new node and restart again.
Thanks,
Matt
Created 11-15-2016 01:25 PM
see if this works for you.
Created 11-15-2016 01:45 PM
The link you reference is good for troubleshooting adding new nodes in NiFi 0.x and HDF 1.x versions of NiFi which had an NCM. NiFi 1.x and HDF 2.x version are very different now and do not have an NCM.
Created 11-15-2016 01:31 PM
@Karthik Narayanan thanks for the url.
They said about templates, what are the templates in the nifi directories ?
I've removed only conf/flow.xml.gz
Created 11-15-2016 01:40 PM
@mayki wogno the thread also mentions deleting a flow.tar file, did you try that.
Created 11-15-2016 01:43 PM
The reported error message is a little misleading, but the key to your issue is reported in the "Caused by" portion of the stack trace:
Causedby: org.apache.nifi.controller.UninheritableFlowException:ProposedAuthorizerisnot inheritable by the flow controller because of Authorizer differences:ProposedAuthorizationsdonot match current Authorizations
Verify that the authorizers.xml file on your new node matches the configuration on your other existing connected nodes.
Remove the users.xml and authorizations.xml files from your new node as well be restarting. The new Node should be getting these files from the cluster when it joins. If you continue to see the same error message on restart, try copying the users.xml and authorizations.xml files from one of your existing connected nodes to this new node and restart again.
Thanks,
Matt
Created 11-15-2016 01:44 PM
@Karthik Narayanan : except flow.xml.gz, i don't have file flow.tar
Created 11-15-2016 01:51 PM
It worked to copy users and authorization.
thanks all.
Created 11-15-2016 02:05 PM
dont do the SCP part. the files will get downloaded from the NCM when the node connects to it.