Member since
07-19-2016
88
Posts
13
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1386 | 11-03-2016 05:31 PM | |
| 4100 | 08-22-2016 06:53 PM |
08-25-2018
06:02 AM
@Dan Chaffelson myid file is already present in the required path inside docker container, I doubt there is need to give docker ip's in the connection string, host address properties of zookeeper and nifi properties file. As of now I Have given my host IP with the two ports open on two dockers(8051,8052) so i think it is trying to find it on the host only and not docker containers. Nifi has embedded ZK individually running on each docker instance. Quorum only comes to picture if there is individual ZK nodes set up and we need to specify which is the primary one
... View more
08-24-2018
06:29 PM
Hi, I have a gcp so i had made two dockers on it with nifi installed on it I'm able to run them individually but as soon as i run them as in cluster by changing zookeeper and nifi properties it stops working gives me the below error. I had already cleared my provenance repo but still getting the same error . I was following the link from here. log: 2018-08-24 05:45:20,296 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0 2018-08-24 05:45:20,309 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 108218562150 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2018-08-24 05:45:20,310 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2018-08-24 05:45:20,652 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5a8ef07 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2018-08-24 05:45:20,652 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 4288 records in 38 milliseconds 2018-08-24 05:45:20,677 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5a8ef07 checkpointed with 4288 Records and 0 Swap Files in 24 milliseconds (Stop-the-world time = 6 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 1475996 2018-08-24 05:45:20,746 INFO [main] o.apache.nifi.controller.FlowController Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not set 2018-08-24 05:45:20,779 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@477b1fca{/nifi-api,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-api-1.6.0.war/webapp/,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.6.0.war} org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:452) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:419) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:777) at org.apache.nifi.NiFi.<init>(NiFi.java:157) at org.apache.nifi.NiFi.<init>(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55) ... 33 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 39 common frames omitted Caused by: java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:350) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:55) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:189) at org.apache.nifi.controller.FlowController.<init>(FlowController.java:608) at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:445) at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 46 common frames omitted 2018-08-24 05:45:21,495 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=445ms 2018-08-24 05:45:21,498 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,535 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@49c45057{/nifi-content-viewer,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-content-viewer-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.6.0.war} 2018-08-24 05:45:21,563 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@2c85c878{/nifi-docs,null,AVAILABLE} 2018-08-24 05:45:21,616 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=33ms 2018-08-24 05:45:21,619 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,668 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@47287357{/nifi-docs,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-docs-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.6.0.war} 2018-08-24 05:45:21,717 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=31ms 2018-08-24 05:45:21,751 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,757 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@aa8284e{/,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-error-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.6.0.war} 2018-08-24 05:45:21,785 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@727b060{HTTP/1.1,[http/1.1]}{0.0.0.0:8051} 2018-08-24 05:45:21,785 INFO [main] org.eclipse.jetty.server.Server Started @61390ms 2018-08-24 05:45:21,786 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:452) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:419) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:777) at org.apache.nifi.NiFi.<init>(NiFi.java:157) at org.apache.nifi.NiFi.<init>(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55) ... 33 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 39 common frames omitted Caused by: java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:350) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:55) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:189) at org.apache.nifi.controller.FlowController.<init>(FlowController.java:608) at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:445) at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 46 common frames omitted 2018-08-24 05:45:21,788 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2018-08-24 05:45:21,794 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@727b060{HTTP/1.1,[http/1.1]}{0.0.0.0:8051} 2018-08-24 05:45:21,795 INFO [Thread-1] org.eclipse.jetty.server.session Stopped scavenging
... View more
Labels:
- Labels:
-
Apache NiFi
-
Docker
08-09-2018
01:38 PM
Thanks @Shu
... View more
08-09-2018
06:18 AM
@Matt Burgess Can you add something on it
... View more
08-09-2018
06:15 AM
I'm trying to hit a get request on my elastic search index using JSONelastisearchprocessor in NIFI. Now, for each flow file i have some incoming attributes, based on that i need to generate different get request and store the response somewhere. The list of processors i'm using is as below: Getfile (to read JSON file) Evaluate JSON path (To extract the attributes which i want to use with every get request, PROC_INST_ID_ in this case ) JSON queryelastic search (to hit the request with the below ) PUTfile to store the response Request body {
"query": {
"nested": {
"path": "los",
"query": {
"bool": {
"must": [
{ "match": { "los.${proc_ins_id}":"784525" }},
{ "match": { "los._source.cibilPermission.VALUE_":"1" }}
]
}
}
}
}
}
I can't see the request being genrated and not getting any response instead i'm only getting the value of proc_ins_id as reponse in putfile. Can someone suggest some appropriate way to do this? Attaching relevant screenshots as well for reference.
... View more
Labels:
- Labels:
-
Apache NiFi
05-24-2018
04:37 PM
@shu tried almost all the thing mentioned above but still no luck
... View more
05-23-2018
06:33 PM
Hi, I'm trying to connect to nifi from mysql, so while connecting i'm facing some poolable connection error like My client nifi is set up on docker and mysql is on host that is local. I can see sql workbench able to connect to mysql on 127.0.0.1 and 3306 so it means port is opened. But looking the below error suggest that there is some problem connecting to mysql Any help on this would be appreciated
... View more
Labels:
- Labels:
-
Apache NiFi
03-14-2018
08:47 AM
@Shu Appreciate your efforts.
... View more
03-14-2018
07:10 AM
I can't see the add to reply option so osting it here again @Shu Thanks for such a detailed explanation. Even if i try to import the template with a changed template name during import it still shoes the same error.
... View more