Support Questions

Find answers, ask questions, and share your expertise

Apache nifi - How can I do bulk upload from one database to another without keeping processor running full time.

New Contributor


I have tried several processor like QueryDatabaseTable --> PutDatabaserecord and ExecuteSQL-->Convert AvroToJson -->ConvertJSON to SQL and PUTSQL. One problem I am facing with Execute SQL is that once all record are being fetched and processed, it will again start fetching from beginning. Do we have any solution for this.


Super Guru

We can store state to DistributeMapCache/HDFS/Hive/Hbase and pullled the stored state and use it in ExecuteSQL processor to incrementally pull the data.

Try with this approach mentioned in this link.


In QueryDatabaseTable processor we need to configure max value columns so that nifi stores the state in to QueryDatabaseTable processor.

In next run QueryDatabaseTable pulls only the latest changes from the table.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.