Migrating realtime data from Sql Server DB to MongoDB using Nifi ?

[#1]

Hi,

I'm just a beginner in using NiFI and I'm looking to use it To migrate data from SQL server to MongoDB in real time.

So my question :

Can we migrate data in realtime from Sql Server to MongoDB using Nifi ? Otherwise, can we connect to a CDC (Capture Data Change) in realtime to capture the data change and process them to MogoDB.

Thank you very much for your help.

[#2]

Hi Hejer,
yes you should be able to move data from SQL Server to MongoDB. If this can be done in real time depends on the performance of the SQL Server and MongoDB server.
To get data from SQL Server you'll need the JDBC driver, and there is a MongoDB processor as well for Nifi.

[#3]

Hi Harald,

Thank you so much for your quick response !!

But please, I need more precisions !

As I understand, NiFi can connect to SQL Server using the JDBC driver and then process data to MongoDB. With this solution, we have to check manually the SQL database changes in real time and process them to MongoDB?

So my question, can we, instead, connect NiFi to a CDC tool (a log-based Capture Data Change) to collect the database changes ?

Thank you so much again !

[#4]

you can connect Nifi via a CDC tool like Atunity if needed. If your SQL server is MS SQL, it should be also possible to enable CDC tables. Nifi itself has also the possibility to query changed data from a table. If you have an insert or update timestamp in your table you could use the QueryDatabaseTable processor.

some more hints: https://community.hortonworks.com/questions/88686/change-data-capture-using-nifi-1.html