Hi, I have a gcp so i had made two dockers on it with nifi installed on it I'm able to run them individually but as soon as i run them as in cluster by changing zookeeper and nifi properties it stops working gives me the below error. I had already cleared my provenance repo but still getting the same error . I was following the link from here. log: 2018-08-24 05:45:20,296 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0 2018-08-24 05:45:20,309 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 108218562150 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2018-08-24 05:45:20,310 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2018-08-24 05:45:20,652 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5a8ef07 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2018-08-24 05:45:20,652 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 4288 records in 38 milliseconds 2018-08-24 05:45:20,677 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5a8ef07 checkpointed with 4288 Records and 0 Swap Files in 24 milliseconds (Stop-the-world time = 6 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 1475996 2018-08-24 05:45:20,746 INFO [main] o.apache.nifi.controller.FlowController Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not set 2018-08-24 05:45:20,779 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@477b1fca{/nifi-api,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-api-1.6.0.war/webapp/,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.6.0.war} org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:452) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:419) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:777) at org.apache.nifi.NiFi.<init>(NiFi.java:157) at org.apache.nifi.NiFi.<init>(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55) ... 33 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 39 common frames omitted Caused by: java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:350) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:55) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:189) at org.apache.nifi.controller.FlowController.<init>(FlowController.java:608) at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:445) at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 46 common frames omitted 2018-08-24 05:45:21,495 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=445ms 2018-08-24 05:45:21,498 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,535 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@49c45057{/nifi-content-viewer,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-content-viewer-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.6.0.war} 2018-08-24 05:45:21,563 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@2c85c878{/nifi-docs,null,AVAILABLE} 2018-08-24 05:45:21,616 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=33ms 2018-08-24 05:45:21,619 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,668 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@47287357{/nifi-docs,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-docs-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.6.0.war} 2018-08-24 05:45:21,717 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=31ms 2018-08-24 05:45:21,751 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,757 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@aa8284e{/,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-error-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.6.0.war} 2018-08-24 05:45:21,785 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@727b060{HTTP/1.1,[http/1.1]}{0.0.0.0:8051} 2018-08-24 05:45:21,785 INFO [main] org.eclipse.jetty.server.Server Started @61390ms 2018-08-24 05:45:21,786 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:452) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:419) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:777) at org.apache.nifi.NiFi.<init>(NiFi.java:157) at org.apache.nifi.NiFi.<init>(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55) ... 33 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 39 common frames omitted Caused by: java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:350) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:55) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:189) at org.apache.nifi.controller.FlowController.<init>(FlowController.java:608) at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:445) at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 46 common frames omitted 2018-08-24 05:45:21,788 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2018-08-24 05:45:21,794 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@727b060{HTTP/1.1,[http/1.1]}{0.0.0.0:8051} 2018-08-24 05:45:21,795 INFO [Thread-1] org.eclipse.jetty.server.session Stopped scavenging ... View more

Hi, I'm using a pyspark processor in NIFI 1.6.0 to execute a script and storing the results on the local. I can see the operation is being performed but how do I let my processor know to stop if it has finished the execution of the script. As it keeps on streaming This concerns me more because once the files get moved it starts throwing errors in the logs which are not tempting at all. @Shu Suggestions from you would be helpful as well ... View more

I'm trying to insert data to mysql from flat(.txt) file and i am using the processors list as (Getfile->ReplaceText->PutSQL). I can see the data from my flat file till Getfile processor but as soon it reaches Replacetext the values for the columns shows as ? . I'm using the below Replacement Value in replace text: insert into flowfiles(id,name) values('${id}', '${name}') Data provinence in replace text: insert into flowfiles(id,name) values('', '') Any help on this would be appreciated ... View more

Hi, I'm trying to connect to nifi from mysql, so while connecting i'm facing some poolable connection error like My client nifi is set up on docker and mysql is on host that is local. I can see sql workbench able to connect to mysql on 127.0.0.1 and 3306 so it means port is opened. But looking the below error suggest that there is some problem connecting to mysql Any help on this would be appreciated ... View more

I want to actually build a custom processor in Nifi from a Github Repo(GIT Hub repo link) and i followed the below steps:- I cloned the repo to my Nifi server install maven Ran maven to build the Repo in my Nifi, then copied the NAR file to biin directory of Nifi reading the last points of Build processor link as i didn't wanted to build a default processor given in the link Restarted my Nifi but the processor is not showing up in the NIFI Gui ... View more

Hi, I have been reading a json file and storing in HDFS in normal csv format. I have created my hive table manually from hive currently and i want to do it using nifi . I have read we can do it using Puthiveql processor but i can't find any attribute here in the processor to insert DDl like we get in selectHIVEql processor. ... View more

I'm trying to connect Hive to fetch some tables using pyhs2 in Embedded/Pseudo Mode. I have kept all the properties in pyhs2.connect same except port number which i got from the reference site mentioned here #!/usr/bin/env python import pyhs2 with pyhs2.connect(host='localhost',port=50070,authMechanism="PLAIN", user='root',password='test',database='default') as conn: with conn.cursor() as cur: #Show databases print cur.getDatabases() #Execute query cur.execute("select * from analytical")#Return column info from query print cur.getSchema() #Fetch table results for i in cur.fetch(): print i Finally when i execute it it show me the below ouput and not able to read anything. It was in embedded mode so havn't put the port as 10000 Here. Can someone help me getting this problem solved. vaibhav@vaibhav-Lenovo-G570:/home/hduser$ ./Automation2.py Traceback(most recent call last):File"./Automation2.py", line 4,in<module>with pyhs2.connect(host='localhost',port=50070,authMechanism="PLAIN", user='root',password='test',database='default')as conn:File"/home/vaibhav/.local/lib/python2.7/site-packages/pyhs2/__init__.py", line 7,in connect returnConnection(*args,**kwargs)File"/home/vaibhav/.local/lib/python2.7/site-packages/pyhs2/connections.py", line 46,in __init__ transport.open()File"/home/vaibhav/.local/lib/python2.7/site-packages/pyhs2/cloudera/thrift_sasl.py", line 74,in open status, payload = self._recv_sasl_message()File"/home/vaibhav/.local/lib/python2.7/site-packages/pyhs2/cloudera/thrift_sasl.py", line 95,in _recv_sasl_message payload = self._trans.readAll(length)File"/home/vaibhav/.local/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 60,in readAll chunk = self.read(sz - have)File"/home/vaibhav/.local/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 132,in read message='TSocket read 0 bytes') thrift.transport.TTransport.TTransportException:TSocket read 0 bytes ... View more

I have a file whose structure is like mentioned below: ID,Name,Address 1,"Amrit,kumar",India 2,"Vaibhav,arora",USA 3,"Deepika,kumar",Germany Obviously if i give pigStorage(',') the three Fields will get splitted into 4 and the data spill over. Alternatives : I tried piggy bank jars but the issue still exist and the data still spills.Please find below the script A11 = LOAD 'File.csv.gz' USING org.apache.pig.piggybank.storage.CSVLoader() as (column:type) I tried Replace fucntiion as well i was having 35k rows the change is not gettting take place for all the rows.Any how the data still spills in this case as well.Column value get shifted to next column.Please find below the referred link. how can i ignore " (double quotes) while loading file in PIG? I tried CSVEXCEL Storage and CSV loader as well. Please suggest what are the things that i can do here. I want to have the name value in a single column. ... View more

'm handing JSON arrays in hive.I know that it can be handled with JSON serde as well but i want to do it with get_json_object or json_tuple and the regular expressions. so i was follwing the link http://mechanics.flite.com/blog/2014/04/16/using-explode-and-lateral-view-in-hive/ It gives the Json File like this {"key1":"0","key2":"11.800","info":"[{\"id\":\"1234\",\"v\":\"3\"},{\"id\":\"5678\",\"v\":\"3\"}]","key3":"3"} Why it need to have slashes in front of id and every other value??.The Observation Are like below When we add the split on the regex we got from the above result we get the below things ["{\"id\":\"1234\",\"v\":\"3\"}","{\"id\":\"5678\",\"v\":\"3\"}"] If i remove the slashes in my source Json file ebrything is coming as null.What is the possible way to get out of this. ... View more