i have a .net web app and a mobile app that store their data into one sql server 2008 database . My objective is to create a real time dashboard platform that'll take data pushed from my database in real time and perform analytics and then get them out to dashboards . How can i push data from the sql server database to some big data broker without impacting the database performance ?
I see there could be 2 approaches to this one.
1. One use sqoop to import the dataset from sql server to hdfs and then you can build a dashboard on the same and after that use incremental import to import the changes. (or)
2. Use NiFi with Database processor to get the records from DB and using store that at relevant location.
The problem that i have is that my database can't stand lots of querying , it already works with two applications . I dont want my database to be queried , instead i want it to push data to some messaging system by itself to work in a real time way
Thanks for the information. with the provided info, it seems like you would need to use sql adaptor of some sort to track the changes and push it out. I did come across these articles which can help you out (I guess) and pass in the information to the cluster using somesort of listener which can read the data from the intermediate client/server.