Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

Apache nifi - How can I do bulk upload from one database to another without keeping processor running full time.

New Contributor

Hi,

I have tried several processor like QueryDatabaseTable --> PutDatabaserecord and ExecuteSQL-->Convert AvroToJson -->ConvertJSON to SQL and PUTSQL. One problem I am facing with Execute SQL is that once all record are being fetched and processed, it will again start fetching from beginning. Do we have any solution for this.

1 REPLY 1

Super Guru

We can store state to DistributeMapCache/HDFS/Hive/Hbase and pullled the stored state and use it in ExecuteSQL processor to incrementally pull the data.

Try with this approach mentioned in this link.

-

In QueryDatabaseTable processor we need to configure max value columns so that nifi stores the state in to QueryDatabaseTable processor.

In next run QueryDatabaseTable pulls only the latest changes from the table.