Member since
07-08-2016
260
Posts
44
Kudos Received
10
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3111 | 05-02-2018 06:03 PM | |
6266 | 10-18-2017 04:02 PM | |
2049 | 08-25-2017 08:59 PM | |
2802 | 07-21-2017 08:13 PM | |
10726 | 04-06-2017 09:54 PM |
11-02-2016
04:40 PM
2 Kudos
Hi, I am getting errors while trying to connect to Hive in out Kerberized HADOOP cluster. Here is what i am doing.. Nifi config properties: # kerberos # nifi.kerberos.krb5.file=/etc/krb5.conf # kerberos service principle # nifi.kerberos.service.principal=nifi/ourserver@ourdomain
nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.keytab Configure Controller Service as soon as i enable it i see some warnings.. and getting these errors when i tried to use the connection in SelectHiveQL process 16:33:08 UTC
ERROR
7c2b4a17-f772-1ea7-54c9-99cc4d8dea09
HiveConnectionPool[id=7c2b4a17-f772-1ea7-54c9-99cc4d8dea09] Error getting Hive connection 16:33:08 UTC
ERROR
d3b62ee6-0157-1000-b66f-364970fcfa98
SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://myserver:10000/hdf_moat;principal=hive/myserver@mydomain: GSS initiate failed). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://myserver:10000/hdf_moat;principal=hive/mydomain: GSS initiate failed)
Regards,
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
11-01-2016
05:11 PM
going with WINDOWS server is as per our company policy and i have changed the bootstrap to use 8GB as per the above document and also running on JDK 1.8...dont know whats causing this.
... View more
11-01-2016
04:58 PM
1 Kudo
I just installed latest version of NiFi on a WINDOWS server (24 core 198 GB RAM). when i tried to open up NiFi , it is spinning on the NiFi logo for a long time before loading UI , even after loading the UI , i tried to drag and drop a couple of processors . but they are all acting very very slow.. i literally had to wait 5 secs before the processor popup shows. task manager shows nothing unusual and nothing else is running on the server. any idea on what might be causing this.? Regards,
... View more
Labels:
- Labels:
-
Apache NiFi
10-27-2016
09:44 PM
@mclark @Matt Burgess so in any case does the file needs to be read in to memory before it splits.?? either by lines or by bytes. i was hoping it starts the next process work once it receive first split.? in my case it waited 8 minutes until it split the 10GB file into 1200+ splits. If my files are about 100 GB each (I have 18 such files) I am scared to run the whole flow for all files. I may have to run for each file one by one.?
... View more
10-26-2016
09:56 PM
@mclark @Matt Burgess Thanks a lot to both of you. I got it to work with a smaller file and tried with a 10 GB file. and experiencing performance issues as expected. trying to see which processes I can set "concurrent tasks" and improve performance. the SplitText with 100000 lines took nearly 8 minutes. can I multi thread it.? is it thread aware.? Splitfile where it took 7.42 mins I took this screen shot after 10 mins.. but when I tried 4 concurrent tasks with RouteText , it routed lot of data to Original relation which I found thru data provenance. any advice.?? we will get 100 GB files in reality. I was trying to prove the concept with 10GB file. Regards, Sai
... View more
10-26-2016
06:38 PM
@Matt Burgess , if I have 5K tags it would be difficult to write dynamic properties for all .I am trying to see how I can get the Tag name dynamically from RouteText , the one that I wrote above is not doing it.
... View more
10-26-2016
05:57 PM
@mclark @Matt Burgess, sorry , I didn't see your reply. may be I should use the above approach with splitText. So it would be combination of multiple splitText and RouteText processes.?? how do I extract the tagName (first column) from the text so that I can use that for Merge process.? Regards, Sai
... View more
10-26-2016
05:48 PM
1 Kudo
Hi @mclark, There will be thousands of tagnames do I have to specify those as relationships.? I hope you are talking about the following way which I tried on a smaller file. which it split in to 5 files based on the grouping reg ex. but it routed all the split files in to unmatched relationship. I think that should be fine I can drive my remaining process from Unmatched. now will it work if the file is huge (100GB).? does it need to read the whole file before it splits based on groups.? routeText properties:
... View more