Support Questions
Find answers, ask questions, and share your expertise

how to make sql server database push data to some big data broker without impacting the DB performance ?

New Contributor

i have a .net web app and a mobile app that store their data into one sql server 2008 database . My objective is to create a real time dashboard platform that'll take data pushed from my database in real time and perform analytics and then get them out to dashboards . How can i push data from the sql server database to some big data broker without impacting the database performance ?

5 REPLIES 5

Expert Contributor
@Haitam Dadsi

I see there could be 2 approaches to this one.

1. One use sqoop to import the dataset from sql server to hdfs and then you can build a dashboard on the same and after that use incremental import to import the changes. (or)

2. Use NiFi with Database processor to get the records from DB and using store that at relevant location.

New Contributor

@Mahesh Balakrishnan

The problem that i have is that my database can't stand lots of querying , it already works with two applications . I dont want my database to be queried , instead i want it to push data to some messaging system by itself to work in a real time way

Expert Contributor
@Haitam Dadsi

Thanks for the information. with the provided info, it seems like you would need to use sql adaptor of some sort to track the changes and push it out. I did come across these articles which can help you out (I guess) and pass in the information to the cluster using somesort of listener which can read the data from the intermediate client/server.

https://github.com/christiandelbianco/monitor-table-change-with-sqltabledependency

https://www.codeproject.com/Articles/63568/HTTP-Push-from-SQL-Server-Comet-SQL

@Haitam Dadsi

You can use CDC tools such as Attunity Replicate to push event to Kafka, then consume from Kafka with NiFi and update a realtime dashboard (Solr for instance)

New Contributor

@Abdelkrim Hadjidj

thanks, i think attunity is not open source , i'm looking for an open source solution

; ;