Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Real Time DATA ingest between Oracle and HDFS

Highlighted

Real Time DATA ingest between Oracle and HDFS

New Contributor

We want to refresh the HDFS with Oracle Data real time, I have option to use Oralce Gloden Gate  FLUME however Oralce GG creates lot of redo and it is problmetic to Oracle Performance. Do we have any other option for real time data sync.

2 REPLIES 2

Re: Real Time DATA ingest between Oracle and HDFS

Cloudera Employee

If you need to get your data from Oracle to HDFS in *near real time* then a Golden Gate solution is probably the best option. While you may need some additional redo logging for this to work, it probably results in the minimum overall impact on the database.

 

If you can tolerate some lag in the data appearing in HDFS, so you could setup some jobs to use Sqoop to pull data from Oracle at regular intervals. With the correct indexes and a way for Sqoop to identify new records this could be done quite efficiently.

 

Depending on how the applicaiton that writes the data to Oracle is architected, there could also be an option to change that applicaiton to write to Oracle and also write to Flume or Kafka, but that does require significant applicaiton changes which may not be feasible in your case.

Re: Real Time DATA ingest between Oracle and HDFS

New Contributor

What about ingestion from MS sql since GG does not support ingestion from MS-Sql server.

Any other option if not GG. Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here