Hello, So I'm trying to create a flow wherein one flow will export data from a table to an S3 bucket in avro format and the other flow will import that data into a table. It would be best if we could do an UPSERT, however as there are many tables, was trying to keep the flow as dynamic and small as possible. The schema exists in the avro registry thus the PutDatabaseRecord works perfectly without writing any sql query. Tried creating a flow wherein we could separate each avro record into 1 record and if INSERT fails then update, but with over a million records, that is extremely slow. The last option we have is to TRUNCATE and re-insert all the data, but those get separated into 2 flows, truncate in a separate flow and then insert in a different flow. Has anyone else run into this issue? Would like to know if its possible to combine truncate and insert into one transaction so we are sure there is always data available. Secondly, what is the best way to do an UPSERT if any? The data source doesn't have a timestamp hence it's hard to check what really was created/updated hence the upsert reasoning and the source tables cannot be changed.
... View more
