question Need Help About Apache NiFi in Support Questions

Need Help About Apache NiFi

omeraran — Mon, 10 Jun 2024 09:04:09 GMT

Hello,

I have MsSql DB and Oracle DB. I am gonna try to explain what I want to do. I am fetching data from MsSQL DB than I am inserting that rows to Oracle DB but this structure not dynamic. Because I created TEST table on both side. So I am writing to table name of processor configuration. I want to create dynamic structure so when I start to flow, I need to migrate all database tables with all rows. But at the same time all tables need to create by processor automatically and when new rows came to MsSQL DB, the flow need to insert only new rows. Any one knows how to do that on Nifi 2.0.0-M3 ?

Re: Need Help About Apache NiFi

DianaTorres — Mon, 10 Jun 2024 17:20:05 GMT

@omeraran Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @mburgess who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Need Help About Apache NiFi

MattWho — Mon, 10 Jun 2024 19:44:32 GMT

@omeraran

This use case sounds like a dataflow that would consist of the following:

GenerateTableFetch --> ExecuteSQL --> <any processor you may want to modify, extract, etc content if needed) --> PutDatabaseRecord

GenerateTableFetch will ingest rows from your source MySQL DB and maintain use NiFi state to record the maximum values for records so that it can continue to check for any ingest additional rows added. It generates FlowFiles that contain the SQL queries needed by the ExecuteSQL to fetch the rows. I don't know if your use case requires any manipulation, routing modifying, etc, but if so you would do that next. And finally use the PutDatabaseRecord to write your rows to the Oracle DB.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Need Help About Apache NiFi

DianaTorres — Thu, 13 Jun 2024 21:36:15 GMT

@omeraran Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.