Member since
07-19-2016
88
Posts
13
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
555 | 11-03-2016 05:31 PM | |
1559 | 08-22-2016 06:53 PM |
08-25-2018
06:02 AM
@Dan Chaffelson myid file is already present in the required path inside docker container, I doubt there is need to give docker ip's in the connection string, host address properties of zookeeper and nifi properties file. As of now I Have given my host IP with the two ports open on two dockers(8051,8052) so i think it is trying to find it on the host only and not docker containers. Nifi has embedded ZK individually running on each docker instance. Quorum only comes to picture if there is individual ZK nodes set up and we need to specify which is the primary one
... View more
08-24-2018
06:29 PM
Hi, I have a gcp so i had made two dockers on it with nifi installed on it I'm able to run them individually but as soon as i run them as in cluster by changing zookeeper and nifi properties it stops working gives me the below error. I had already cleared my provenance repo but still getting the same error . I was following the link from here. log: 2018-08-24 05:45:20,296 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0 2018-08-24 05:45:20,309 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 108218562150 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2018-08-24 05:45:20,310 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2018-08-24 05:45:20,652 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5a8ef07 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2018-08-24 05:45:20,652 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 4288 records in 38 milliseconds 2018-08-24 05:45:20,677 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5a8ef07 checkpointed with 4288 Records and 0 Swap Files in 24 milliseconds (Stop-the-world time = 6 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 1475996 2018-08-24 05:45:20,746 INFO [main] o.apache.nifi.controller.FlowController Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not set 2018-08-24 05:45:20,779 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@477b1fca{/nifi-api,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-api-1.6.0.war/webapp/,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.6.0.war} org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:452) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:419) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:777) at org.apache.nifi.NiFi.<init>(NiFi.java:157) at org.apache.nifi.NiFi.<init>(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55) ... 33 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 39 common frames omitted Caused by: java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:350) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:55) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:189) at org.apache.nifi.controller.FlowController.<init>(FlowController.java:608) at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:445) at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 46 common frames omitted 2018-08-24 05:45:21,495 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=445ms 2018-08-24 05:45:21,498 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,535 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@49c45057{/nifi-content-viewer,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-content-viewer-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.6.0.war} 2018-08-24 05:45:21,563 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@2c85c878{/nifi-docs,null,AVAILABLE} 2018-08-24 05:45:21,616 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=33ms 2018-08-24 05:45:21,619 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,668 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@47287357{/nifi-docs,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-docs-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.6.0.war} 2018-08-24 05:45:21,717 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=31ms 2018-08-24 05:45:21,751 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath 2018-08-24 05:45:21,757 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@aa8284e{/,file:///nifi/nifi-1.6.0/work/jetty/nifi-web-error-1.6.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.6.0.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.6.0.war} 2018-08-24 05:45:21,785 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@727b060{HTTP/1.1,[http/1.1]}{0.0.0.0:8051} 2018-08-24 05:45:21,785 INFO [main] org.eclipse.jetty.server.Server Started @61390ms 2018-08-24 05:45:21,786 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller. at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:452) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:419) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:777) at org.apache.nifi.NiFi.<init>(NiFi.java:157) at org.apache.nifi.NiFi.<init>(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55) ... 33 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084) at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 39 common frames omitted Caused by: java.lang.IllegalArgumentException: ./state/zookeeper/myid file is missing at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:350) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:55) at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:189) at org.apache.nifi.controller.FlowController.<init>(FlowController.java:608) at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:445) at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 46 common frames omitted 2018-08-24 05:45:21,788 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2018-08-24 05:45:21,794 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@727b060{HTTP/1.1,[http/1.1]}{0.0.0.0:8051} 2018-08-24 05:45:21,795 INFO [Thread-1] org.eclipse.jetty.server.session Stopped scavenging
... View more
Labels:
08-09-2018
01:38 PM
Thanks @Shu
... View more
08-09-2018
06:18 AM
@Matt Burgess Can you add something on it
... View more
08-09-2018
06:15 AM
I'm trying to hit a get request on my elastic search index using JSONelastisearchprocessor in NIFI. Now, for each flow file i have some incoming attributes, based on that i need to generate different get request and store the response somewhere. The list of processors i'm using is as below: Getfile (to read JSON file) Evaluate JSON path (To extract the attributes which i want to use with every get request, PROC_INST_ID_ in this case ) JSON queryelastic search (to hit the request with the below ) PUTfile to store the response Request body {
"query": {
"nested": {
"path": "los",
"query": {
"bool": {
"must": [
{ "match": { "los.${proc_ins_id}":"784525" }},
{ "match": { "los._source.cibilPermission.VALUE_":"1" }}
]
}
}
}
}
}
I can't see the request being genrated and not getting any response instead i'm only getting the value of proc_ins_id as reponse in putfile. Can someone suggest some appropriate way to do this? Attaching relevant screenshots as well for reference.
... View more
Labels:
07-30-2018
06:55 AM
Hi, I'm using a pyspark processor in NIFI 1.6.0 to execute a script and storing the results on the local. I can see the operation is being performed but how do I let my processor know to stop if it has finished the execution of the script. As it keeps on streaming This concerns me more because once the files get moved it starts throwing errors in the logs which are not tempting at all. @Shu Suggestions from you would be helpful as well
... View more
Labels:
05-24-2018
04:37 PM
@shu tried almost all the thing mentioned above but still no luck
... View more
05-24-2018
04:35 PM
@Shu any comment on this?
... View more
05-24-2018
04:34 PM
I'm trying to insert data to mysql from flat(.txt) file and i am using the processors list as (Getfile->ReplaceText->PutSQL).
I can see the data from my flat file till Getfile processor but as soon it reaches Replacetext the values for the columns shows as ? .
I'm using the below Replacement Value in replace text: insert into flowfiles(id,name) values('${id}', '${name}') Data provinence in replace text: insert into flowfiles(id,name) values('', '') Any help on this would be appreciated
... View more
Labels:
05-23-2018
06:33 PM
Hi, I'm trying to connect to nifi from mysql, so while connecting i'm facing some poolable connection error like My client nifi is set up on docker and mysql is on host that is local. I can see sql workbench able to connect to mysql on 127.0.0.1 and 3306 so it means port is opened. But looking the below error suggest that there is some problem connecting to mysql Any help on this would be appreciated
... View more
Labels:
03-18-2018
07:08 AM
It worked. Thanks
... View more
03-18-2018
07:07 AM
It worked. Thanks
... View more
03-16-2018
01:34 PM
@Shu Any suggestions from you?
... View more
03-16-2018
01:33 PM
1 Kudo
I want to actually build a custom processor in Nifi from a Github Repo(GIT Hub repo link) and i followed the below steps:-
I cloned the repo to my Nifi server install maven
Ran maven to build the Repo in my Nifi, then copied the NAR file to biin directory of Nifi reading the last points of Build processor link as i didn't wanted to build a default processor given in the link Restarted my Nifi but the processor is not showing up in the NIFI Gui
... View more
Labels:
03-14-2018
08:47 AM
@Shu Appreciate your efforts.
... View more
03-14-2018
07:23 AM
Hi, I have been reading a json file and storing in HDFS in normal csv format. I have created my hive table manually from hive currently and i want to do it using nifi . I have read we can do it using Puthiveql processor but i can't find any attribute here in the processor to insert DDl like we get in selectHIVEql processor.
... View more
Labels:
03-14-2018
07:10 AM
I can't see the add to reply option so osting it here again @Shu Thanks for such a detailed explanation. Even if i try to import the template with a changed template name during import it still shoes the same error.
... View more
03-13-2018
02:30 PM
@ShuAccepted. It would be good if you can clarify what exactly happens at the background when we check/un-check relation checkboxes. I couldn't find a good description online. Moreover while importing saved XML template it shows error that same template already exist. I think it has something related to PCG. Any comments on this??
... View more
03-13-2018
12:10 PM
Add your reply is not working so have to post it as answer @Shu Please find below the screenshot. I can see there is something different in the things highlighted in you image and mine. Can you suggest the changes i need to make?
... View more
03-13-2018
10:36 AM
Input file is present in the attachment section. Coudn't add it to the main setion.
... View more
03-13-2018
10:23 AM
1 Kudo
Hi,
I have my JSON file like mentioned below:
The flow for conversion for json to csv : 1. Get the file 2. Split json 3. Evaluate json path 4. Replace text I want to see values in csv format of memberid and loanamnt and 'm getting blank for the two columns like ""."" Can someone help me in this?? Attached are the Screenshots of the processor properties.
... View more
Labels:
07-05-2017
07:31 PM
But I have seen people using it even in 4gb. Should I try some older versions
... View more
07-05-2017
07:24 PM
Hi , I have given 4gb dedicated ram to Virtual box but still the screen is stuck as shown below. Any suggestions, i have total of 8 gb ram i3 2nd genration laptop
... View more
Labels:
06-24-2017
07:33 PM
I'm trying to connect Hive to fetch some tables using pyhs2 in Embedded/Pseudo Mode. I have kept all the properties in pyhs2.connect same except port number which i got from the reference site mentioned here #!/usr/bin/env python
import pyhs2
with pyhs2.connect(host='localhost',port=50070,authMechanism="PLAIN", user='root',password='test',database='default') as conn:
with conn.cursor() as cur: #Show databases
print cur.getDatabases() #Execute query
cur.execute("select * from analytical")#Return column info from query
print cur.getSchema()
#Fetch table results
for i in cur.fetch(): print i Finally when i execute it it show me the below ouput and not able to read anything. It was in embedded mode so havn't put the port as 10000 Here. Can someone help me getting this problem solved. vaibhav@vaibhav-Lenovo-G570:/home/hduser$ ./Automation2.py
Traceback(most recent call last):File"./Automation2.py", line 4,in<module>with pyhs2.connect(host='localhost',port=50070,authMechanism="PLAIN", user='root',password='test',database='default')as conn:File"/home/vaibhav/.local/lib/python2.7/site-packages/pyhs2/__init__.py", line 7,in connect
returnConnection(*args,**kwargs)File"/home/vaibhav/.local/lib/python2.7/site-packages/pyhs2/connections.py", line 46,in __init__
transport.open()File"/home/vaibhav/.local/lib/python2.7/site-packages/pyhs2/cloudera/thrift_sasl.py", line 74,in open
status, payload = self._recv_sasl_message()File"/home/vaibhav/.local/lib/python2.7/site-packages/pyhs2/cloudera/thrift_sasl.py", line 95,in _recv_sasl_message
payload = self._trans.readAll(length)File"/home/vaibhav/.local/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 60,in readAll
chunk = self.read(sz - have)File"/home/vaibhav/.local/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 132,in read
message='TSocket read 0 bytes')
thrift.transport.TTransport.TTransportException:TSocket read 0 bytes
... View more
Labels:
05-12-2017
05:24 PM
Number of mappers involved in a job= Number of input splits and number of input splits depends on your block size and file size .If file size is 256 mb and block size is 128mb it will involve 2mappers. @Bala Vignesh N V
... View more
05-10-2017
03:06 PM
@Greg Keysi cannot alter my file here not even temp file,permission issues.
... View more
05-10-2017
07:48 AM
1 Kudo
I have a file whose structure is like mentioned below: ID,Name,Address 1,"Amrit,kumar",India 2,"Vaibhav,arora",USA 3,"Deepika,kumar",Germany Obviously if i give pigStorage(',') the three Fields will get splitted into 4 and the data spill over. Alternatives :
I tried piggy bank jars but the issue still exist and the data still spills.Please find below the script
A11 = LOAD 'File.csv.gz' USING org.apache.pig.piggybank.storage.CSVLoader() as (column:type) I tried Replace fucntiion as well i was having 35k rows the change is not gettting take place for all the rows.Any how the data still spills in this case as well.Column value get shifted to next column.Please find below the referred link.
how can i ignore " (double quotes) while loading file in PIG? I tried CSVEXCEL Storage and CSV loader as well. Please suggest what are the things that i can do here. I want to have the name value in a single column.
... View more
Labels:
01-25-2017
07:43 AM
Hi All, I have my table structure and in hive like below . Create table hv (x int,y int,z int ,w int ) PIGLATIN used to load the table A = LOAD 'XYZ.CSV.GZ' using pigstorage AS (x,y,w) this time Z col is missing from the file so i wan to keep the value of z as null in PIGLATIN and i donot want to change my hive table structure .How to generate the values as null for this column in pig??
... View more
- Tags:
- Data Processing
- hiveserver2
- Mapreduce
- Pig
- Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive
Labels:
01-15-2017
06:12 PM
'm handing JSON arrays in hive.I know that it can be handled with
JSON serde as well but i want to do it with get_json_object or
json_tuple and the regular expressions. so i was follwing the link http://mechanics.flite.com/blog/2014/04/16/using-explode-and-lateral-view-in-hive/ It gives the Json File like this
{"key1":"0","key2":"11.800","info":"[{\"id\":\"1234\",\"v\":\"3\"},{\"id\":\"5678\",\"v\":\"3\"}]","key3":"3"}
Why it need to have slashes in front of id and every other value??.The Observation Are like below When we add the split on the regex we got from the above result we get the below things
["{\"id\":\"1234\",\"v\":\"3\"}","{\"id\":\"5678\",\"v\":\"3\"}"] If i remove the slashes in my source Json file ebrything is coming as null.What is the possible way to get out of this.
... View more
Labels: