Hi, I have a basic etl flow (screenshot below) where i'm trying to start loading a full table from db to another, and then incrementally load any changes to the table based on an updated date. It runs, but part of it is slow. I think the converting json to sql is a bottle neck, as the table has 2M+ records, and they're queued up, but it can only create insert statements so fast, and then inserts to redshift (the target) are not the fastest.
If you look closely at screenshot below right before the convertjsontosql, can see the queue of 3gb (which is about the size of the table), waiting to be converted I believe. Read some of the articles, and set the translate field names to false. also set ignore unmatched columns and fields.
So the initial loading of data is a bit slow. Any suggestions, workarounds, ideas are appeciated to have a process that i know i can start from scratch, load a table fully and then keep loading? or maybe I do have to do one initial large load somehow, and then start the incremental process? I know copy commands in redshift are way better, perhaps i have to do that first, and then start incremental? screen-shot-2018-08-20-at-104837-am.png Thanks,
Ron
... View more
