Member since
05-30-2025
4
Posts
0
Kudos Received
0
Solutions
06-05-2025
12:18 PM
I need to copy a MS SQL table with about 9 million records to a Clickhouse database. I did setup a QueryDatabaseTable processor to pull the table from the SQL database and a PutDatabaseRecord processor to push the records to Clickhouse db. As long as the flowfile from the QueryDatabaseTable processor has less records than the setting of Batch Size in the PutDatabaseRecord everything works fine. But when I have more records the PutDatabaseRecord creates multiple batches. When I pull 30.000 records from my source table and the Batch Size is set to 10.000 I end up with 60.000 records in the destination. Taking a look at the debug information from the PutDatabaseRecord it shows 1 insert and 3 insert batches. When I pull all 9.5 million records I end up with 650 million records in the destination. Any idea ?
... View more
Labels:
- Labels:
-
Apache NiFi
06-04-2025
11:01 AM
@NadirHamburg Can you elaborate on what you are looking for from Cloudera Support? zit seems you are asking a question related to xml storage files which is not related to any of our Cloudera Products. Did you mean to submit this questions to Microsoft? Thank you
... View more
05-31-2025
03:04 AM
This indeed fixed the issue. Thanks a lot !
... View more