Apache nifi - How can I do bulk upload from one database to another without keeping processor running full time.

[#1]

Hi,

I have tried several processor like QueryDatabaseTable --> PutDatabaserecord and ExecuteSQL-->Convert AvroToJson -->ConvertJSON to SQL and PUTSQL. One problem I am facing with Execute SQL is that once all record are being fetched and processed, it will again start fetching from beginning. Do we have any solution for this.

[#2]

We can store state to DistributeMapCache/HDFS/Hive/Hbase and pullled the stored state and use it in ExecuteSQL processor to incrementally pull the data.

Try with this approach mentioned in this link.

-

In QueryDatabaseTable processor we need to configure max value columns so that nifi stores the state in to QueryDatabaseTable processor.

In next run QueryDatabaseTable pulls only the latest changes from the table.