That's the redshift connection: That's the mySQL connection: I do understand that INSERTs on redshift might be slow and that's why people are using COPY command but 100 rows in 5min is weird. I know that for BigQuery for example there's a dedicated processor in order to load data but couldn't find anything for RedShift so that's why I am using putDatabaseRecord. Edit - After I loaded 100 rows I exported the same 100 rows into an INSERT SQL statement and run it directly on the redshift, this task took 1.1 second. Edit2 - Since im the kind of guy that loves doing my job and to find solution I found this - reWriteBatchedInserts=true;reWriteBatchedInsertsSize=102400; I found this on some reddit post by chance, this is working way better now but I couldn't find some details regard those parameters to make them optimized and still that's not the best (still little bit slow). I am not sure that's the solution so i'm still waiting for your input as well. Also, if I would like to change it to drop the Avro/flow files into S3 and then execute COPY command from NIFI is it possible?
... View more
