Member since
07-09-2018
19
Posts
4
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5334 | 08-21-2018 01:19 PM | |
897 | 08-20-2018 01:39 AM | |
1166 | 07-12-2018 08:26 PM | |
968 | 07-09-2018 07:07 PM |
08-24-2018
03:58 PM
Thanks Shu, that seems to work! is that expression language, or is that basically doing regular sql functions on the db? Thanks!
... View more
08-23-2018
08:46 PM
1 Kudo
Hi, I'm getting some data from a pg database, and it has 2 "Text" types, which I think are my problem. When I go to insert those to a redshift table, i'm getting a jdbc error, "given type does not match given object". I think i need to use nifi expression language to convert the text column toString, so it will work with varchar, but i'm not sure. any help appreciated. Thanks, Ron
... View more
Labels:
- Labels:
-
Apache NiFi
08-21-2018
04:13 PM
I actually think the pull is not my problem. its inserts to redshift, which are slow. think i need to put this data in files, and do copy commands to redshift. guess i need to move to saving this data in files, and then copy/load. Or i guess i can manually load tables, and then set an initial max value column, to have the nifi job start from there, and continue updating. These large tables are challenging.
... View more
08-21-2018
02:15 PM
thanks, was just reading something about GTF. much appreciated,
... View more
08-21-2018
01:19 PM
hi @Matt Burgess, i was just reading querydatabasetable's spec, and didn't realize it says its to be run on the primary node only, is that correct? thanks https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html
... View more
08-20-2018
08:46 PM
My jvm heap usage is really high. 85%, and I have fairly large servers. How can I clear these queues and cleanup space? thanks
... View more
08-20-2018
06:25 PM
Hi, nevermind on that, i just created a new process group and tried again. its running now, but still seems slow. the putdbrecord has 3 pending actions, but no other stats yet (eg, in/out). 1.79gb pending in the queue. I set my output batch size of querydbtable to 5,000. i also set fetch size limit and max rows per flowfile to 5k too. does that make sense? thanks
... View more
08-20-2018
05:40 PM
Hey @Matt Burgess, i made a new process group with this, but forgot to use the output batch size property. so i stopped it, changed, and am trying to clear queue and restart, but its stuck in the queue. I've had this happen a few times. any suggestions here? i can't stop, play, terminate or anything. When i do hit empty, it says zero out of 3 flowfiles were removed, like it can't remove them. The processor after the queue has no option to start it. Thanks screen-shot-2018-08-20-at-14100-pm.png
... View more
08-20-2018
03:44 PM
Thanks Matt! will give it a shot
... View more
08-20-2018
02:59 PM
Hi, I have a basic etl flow (screenshot below) where i'm trying to start loading a full table from db to another, and then incrementally load any changes to the table based on an updated date. It runs, but part of it is slow. I think the converting json to sql is a bottle neck, as the table has 2M+ records, and they're queued up, but it can only create insert statements so fast, and then inserts to redshift (the target) are not the fastest.
If you look closely at screenshot below right before the convertjsontosql, can see the queue of 3gb (which is about the size of the table), waiting to be converted I believe. Read some of the articles, and set the translate field names to false. also set ignore unmatched columns and fields.
So the initial loading of data is a bit slow. Any suggestions, workarounds, ideas are appeciated to have a process that i know i can start from scratch, load a table fully and then keep loading? or maybe I do have to do one initial large load somehow, and then start the incremental process? I know copy commands in redshift are way better, perhaps i have to do that first, and then start incremental? screen-shot-2018-08-20-at-104837-am.png Thanks,
Ron
... View more
Labels:
- Labels:
-
Apache NiFi