Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache nifi - How can I do bulk upload from one database to another without keeping processor running full time.

Apache nifi - How can I do bulk upload from one database to another without keeping processor running full time.

New Contributor

Hi,

I have tried several processor like QueryDatabaseTable --> PutDatabaserecord and ExecuteSQL-->Convert AvroToJson -->ConvertJSON to SQL and PUTSQL. One problem I am facing with Execute SQL is that once all record are being fetched and processed, it will again start fetching from beginning. Do we have any solution for this.

1 REPLY 1
Highlighted

Re: Apache nifi - How can I do bulk upload from one database to another without keeping processor running full time.

Super Guru

We can store state to DistributeMapCache/HDFS/Hive/Hbase and pullled the stored state and use it in ExecuteSQL processor to incrementally pull the data.

Try with this approach mentioned in this link.

-

In QueryDatabaseTable processor we need to configure max value columns so that nifi stores the state in to QueryDatabaseTable processor.

In next run QueryDatabaseTable pulls only the latest changes from the table.

Don't have an account?
Coming from Hortonworks? Activate your account here