Skip to main content
omeraran
omeraran
Silver ⭐️
June 10, 2024
Question

Need Help About Apache NiFi

Hello,

I have MsSql DB and Oracle DB. I am gonna try to explain what I want to do. I am fetching data from MsSQL DB than I am inserting that rows to Oracle DB but this structure not dynamic. Because I created TEST table on both side. So I am writing to table name of processor configuration. I want to create dynamic structure so when I start to flow, I need to migrate all database tables with all rows. But at the same time all tables need to create by processor automatically and when new rows came to MsSQL DB, the flow need to insert only new rows. Any one knows how to do that on Nifi 2.0.0-M3 ?

    3 replies

    DianaTorres
    Community Manager
    DianaTorres
    Community Manager
    June 10, 2024

    @omeraran Welcome to the Cloudera Community!

    To help you get the best possible solution, I have tagged our NiFi experts @MattWho @mburgess  who may be able to assist you further.

    Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

    Translate
    Regards, Diana Torres, Senior Community Moderator
    MattWho
    Titanium ⭐️⭐️⭐️
    MattWho
    Titanium ⭐️⭐️⭐️
    June 10, 2024

    @omeraran 

    This use case sounds like a dataflow that would consist of the following:

    GenerateTableFetch --> ExecuteSQL --> <any processor you may want to modify, extract, etc content if needed) --> PutDatabaseRecord

    GenerateTableFetch will ingest rows from your source MySQL DB and maintain use NiFi state to record the maximum values for records so that it can continue to check for any ingest additional rows added.   It generates FlowFiles that contain the SQL queries needed by the ExecuteSQL to fetch the rows.   I don't know if your use case requires any manipulation, routing modifying, etc, but if so you would do that next.  And finally use the PutDatabaseRecord to write your rows to the Oracle DB. 

    Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

    Thank you,
    Matt



    Translate
    DianaTorres
    Community Manager
    DianaTorres
    Community Manager
    June 13, 2024

    @omeraran Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

    Translate
    Regards, Diana Torres, Senior Community Moderator
    Terms & ConditionsAccessibility statement