question Re: Getting Duplicate records in MySQL database in Archives of Support Questions (Read Only)

Getting Duplicate records in MySQL database

ramesh_ganginen — Sun, 18 Aug 2019 10:07:41 GMT

I'm trying to copy the data from one database(MySQL) to another database(MySQL). Below is the flow.

GenerateTableFetch Partition Size : 1500 records. Total 3200 records so it's creating 3 files as shown in the picture.

I'm able to read the data from the source database(avro format) and convert the data to json(using AvrotoJSON processor). Now I'm trying to load the JSON data into MySQL table using ConvertJSONTOSQL Pricessor. In the source table I have 3200

records, however in my destination table I'm getting 10000+ records. I didn't get where I did mis configuration so that it's inserting duplicate records into destination table. Below are screenshots of the individual processors.

Any suggestions please.

Re: Getting Duplicate records in MySQL database

Shu_ashu — Sun, 18 Aug 2019 10:07:28 GMT

@User 805

You need to autoterminate original relationship in ConvertJSONtoSQL processor, now original relationship is loop back into same processor again which is causing all the duplicates in mysql database.

How to Autoterminate original relationship:-

clear the queue first.
delete the original relationship
right click on processor
goto setting tab
click on checkbox at original relationship as shown in the below screenshot

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

Re: Getting Duplicate records in MySQL database

ramesh_ganginen — Wed, 07 Mar 2018 20:36:49 GMT

Hi @Shu,

It worked like charm. Thanks a lot for the quick response.