Posts: 72
Registered: ‎01-19-2017

streaming transaction data from oracle

[ Edited ]

Hi All,


       Is there any way to stream data from oracle table to hdfs in near real time.

Posts: 7
Registered: ‎10-13-2017

Re: streaming transaction data from oracle

I'm new to Hadoop so I use StreamSets to transfer data from Oracle to hdfs. This is assuming there are PKs and Timestamps for the offsets. Caveat is if the buffer is not complete, your records are not real-time until that happens.

The other option is to write a Spark job, include JDBC driver, and write the file to HDFS, see this for a post written in 2015 that can be rewritten