Member since
05-10-2016
303
Posts
35
Kudos Received
0
Solutions
11-23-2016
01:20 PM
Hi all, I'll trying to make a dataflow as explain in this tuto : site-to-site it explains about ListHDFS and FetchHDFS It is possible to use RPG inside the local cluster (I have one cluster with 3 nodes) ? Why I can not use RPG and input port in side the specific process group but I must to create dataflow in the top level.
... View more
Labels:
- Labels:
-
Apache NiFi
11-15-2016
01:51 PM
It worked to copy users and authorization. thanks all.
... View more
11-15-2016
01:44 PM
@Karthik Narayanan : except flow.xml.gz, i don't have file flow.tar
... View more
11-15-2016
01:31 PM
@Karthik Narayanan thanks for the url. They said about templates, what are the templates in the nifi directories ? I've removed only conf/flow.xml.gz
... View more
11-15-2016
01:19 PM
Hello, I'm trying to add new nifi node to my cluster. I have fresh install on this new node so not has flow.xml.gz not templates. zookeeper.properties, nifi.properties are same on every nodes. When I'm trying started new nifi node, it seems that the node is correctly connect to zookeeper server but it has an issue with flow file or authorization. 2016-11-15 14:05:01,809 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE. Building now...
2016-11-15 14:05:02,026 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded 96 properties from system properties and environment variables
2016-11-15 14:05:02,027 INFO [main] o.a.nifi.util.FileBasedVariableRegistry Loaded a total of 96 properties. Including precedence overrides effective accessible registry key size is 96
2016-11-15 14:05:02,144 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 0 records
2016-11-15 14:05:02,150 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0
2016-11-15 14:05:02,158 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 21913605242 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size
2016-11-15 14:05:02,159 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false
2016-11-15 14:05:02,324 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@41b2123e finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates
2016-11-15 14:05:02,325 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 6 milliseconds
2016-11-15 14:05:02,361 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@41b2123e checkpointed with 0 Records and 0 Swap Files in 35 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 5 millis), max Transaction ID -1
2016-11-15 14:05:02,458 INFO [main] o.a.n.c.s.server.ZooKeeperStateServer Starting Embedded ZooKeeper Peer
2016-11-15 14:05:02,680 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected...
2016-11-15 14:05:02,855 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
2016-11-15 14:05:02,973 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-15 14:05:03,034 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:host.name=nifi002
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.version=1.8.0_111
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.vendor=Oracle Corporation
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64/jre
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.class.path=/var/opt/hosting/nifi/./conf:/var/opt/hosting/nifi/./lib/commons-lang3-3.4.jar:/var/opt/hosting/nifi/./lib/logback-core-1.1.3.jar:/var/opt/hosting/nifi/./lib/log4j-over-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/nifi-properties-loader-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-runtime-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-api-1.0.0.jar:/var/opt/hosting/nifi/./lib/jcl-over-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/logback-classic-1.1.3.jar:/var/opt/hosting/nifi/./lib/jul-to-slf4j-1.7.12.jar:/var/opt/hosting/nifi/./lib/bcprov-jdk15on-1.54.jar:/var/opt/hosting/nifi/./lib/nifi-documentation-1.0.0.jar:/var/opt/hosting/nifi/./lib/slf4j-api-1.7.12.jar:/var/opt/hosting/nifi/./lib/nifi-framework-api-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-nar-utils-1.0.0.jar:/var/opt/hosting/nifi/./lib/nifi-properties-1.0.0.jar
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.io.tmpdir=/tmp
2016-11-15 14:05:03,036 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:java.compiler=<NA>
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.name=Linux
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.arch=amd64
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:os.version=2.6.32-642.4.2.el6.x86_64
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.name=root
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.home=/root
2016-11-15 14:05:03,037 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Server environment:user.dir=/var/opt/hosting/nifi
2016-11-15 14:05:03,044 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir ./state/zookeeper/version-2 snapdir ./state/zookeeper/version-2
2016-11-15 14:05:03,051 INFO [main] o.apache.nifi.controller.FlowController The Election for Cluster Coordinator has already begun (Leader is nifi011:11443). Will not register to be elected for this role until after connecting to the cluster and inheriting the cluster's flow.
2016-11-15 14:05:03,052 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election.
2016-11-15 14:05:03,053 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
2016-11-15 14:05:03,060 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election.
2016-11-15 14:05:03,061 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started
2016-11-15 14:05:03,061 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started
2016-11-15 14:05:03,069 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-15 14:05:03,073 INFO [QuorumPeer[myid=2]/0.0.0.0:2181] o.a.z.server.persistence.FileTxnSnapLog Snapshotting: 0x800000028 to ./state/zookeeper/version-2/snapshot.800000028
2016-11-15 14:05:03,109 INFO [SyncThread:2] o.a.z.server.persistence.FileTxnLog Creating new log file: log.800000029
2016-11-15 14:05:03,126 WARN [main] o.eclipse.jetty.util.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlets.GzipFilter
2016-11-15 14:05:03,126 WARN [main] org.eclipse.jetty.servlets.GzipFilter GzipFilter is deprecated. Use GzipHandler
2016-11-15 14:05:03,129 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@166a5659{/nifi-api,file:///var/opt/hosting/nifi/work/jetty/nifi-web-api-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.0.0.war}
2016-11-15 14:05:03,822 INFO [main] /nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath
2016-11-15 14:05:03,846 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@65cc3902{/nifi-content-viewer,file:///var/opt/hosting/nifi/work/jetty/nifi-web-content-viewer-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.0.0.war}
2016-11-15 14:05:03,847 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@a08e41b{/nifi-docs,null,AVAILABLE}
2016-11-15 14:05:03,904 INFO [main] /nifi-docs No Spring WebApplicationInitializer types detected on classpath
2016-11-15 14:05:03,906 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@e08d871{/nifi-docs,file:///var/opt/hosting/nifi/work/jetty/nifi-web-docs-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.0.0.war}
2016-11-15 14:05:03,961 INFO [main] / No Spring WebApplicationInitializer types detected on classpath
2016-11-15 14:05:03,963 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@7962c1d5{/,file:///var/opt/hosting/nifi/work/jetty/nifi-web-error-1.0.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.0.0.war}
2016-11-15 14:05:03,968 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@377cbdae(nifi-key,h=[nifi002],w=[]) for SslContextFactory@37a939bc(file:///var/opt/hosting/nifi/conf/keystore.jks,file:///var/opt/hosting/nifi/conf/truststore.jks)
2016-11-15 14:05:03,981 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@37aace65{SSL,[ssl, http/1.1]}{nifi002:9443}
2016-11-15 14:05:03,982 INFO [main] org.eclipse.jetty.server.Server Started @43215ms
2016-11-15 14:05:05,035 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...
2016-11-15 14:05:05,049 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 11443
2016-11-15 14:05:05,160 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: nifi002:9443
2016-11-15 14:05:05,165 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at nifi011:11443; will use this address for sending heartbeat messages
2016-11-15 14:05:05,563 INFO [Process Cluster Protocol Request-1] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi002:9443 changed from null to NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28]
2016-11-15 14:05:05,568 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 99d259cd-736b-46da-9c8a-20419f1a31f8 (type=NODE_STATUS_CHANGE, length=1182 bytes) from nifi011 in 149 millis
2016-11-15 14:05:05,653 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {nifi002:9443=NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28]} to {nifi012:9443=NodeConnectionStatus[nodeId=nifi012:9443, state=CONNECTED, updateId=17], nifi013:9443=NodeConnectionStatus[nodeId=nifi013:9443, state=CONNECTED, updateId=18], nifi014:9443=NodeConnectionStatus[nodeId=nifi014:9443, state=CONNECTED, updateId=19], nifi001:9443=NodeConnectionStatus[nodeId=nifi001:9443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow., updateId=26], nifi011:9443=NodeConnectionStatus[nodeId=nifi011:9443, state=CONNECTED, updateId=20], nifi002:9443=NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28]}
2016-11-15 14:05:05,662 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) [nifi-jetty-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.<init>(NiFi.java:152) [nifi-runtime-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.main(NiFi.java:243) [nifi-runtime-1.0.0.jar:1.0.0]
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857) ~[nifi-framework-core-1.0.0.jar:1.0.0]
... 4 common frames omitted
2016-11-15 14:05:05,663 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator nifi002:9443 requested disconnection from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
2016-11-15 14:05:05,664 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi002:9443 changed from NodeConnectionStatus[nodeId=nifi002:9443, state=CONNECTING, updateId=28] to NodeConnectionStatus[nodeId=nifi002:9443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow., updateId=28]
2016-11-15 14:05:05,803 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for nifi002:9443 -- Node disconnected from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
2016-11-15 14:05:05,816 ERROR [Shutdown Cluster Coordinator] o.a.z.server.NIOServerCnxnFactory Thread Thread[Shutdown Cluster Coordinator,5,main] died
java.lang.NullPointerException: null
thanks for helps
... View more
Labels:
- Labels:
-
Apache NiFi
11-14-2016
10:48 AM
Hello all, I'm trying to know how my cluster is used by own customers. Do you know if it is possible to add a field like application_id to hdfs-audit.log ? thanks
... View more
Labels:
- Labels:
-
Apache Hadoop
11-07-2016
06:43 PM
Hi All, When putHdfs Process with Data what happen if the Nifi node crash or if fail to write to HDFS ? It is possible to rerun a specific flowfile ? Regards
... View more
Labels:
- Labels:
-
Apache NiFi
11-02-2016
10:28 AM
1 Kudo
Hi all, I wonder to send extract Attribute to ListenHTTP Do you know how I write the rule from : HTTP Headers to receive as Attributes (Regex) || Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes My curl command : curl -v --cacert nifi-cert.pem --cert ./client.p12 -X POST -H 'Content-Type: application/json' -H 'vartest1: v1' -F file=@/var/opt/hosting/log/flume/flume-a1.log.10 https://nifi012:10001/contentListener How I can extract vartest1 as an attribute. Many thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
10-31-2016
09:26 AM
@mclark I've found that was wrong in my config. Before changing storepass for keystore.jks. I need to change keypasswd for private key first So Step 1 : keytool -keypasswd -alias [Alias name for private key] -keystore [path to key store] Step 2 : keytool -storepasswd -new <new_storepass> -keystore keystore.jks
Now It works Thanks @mclark
... View more