Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

RDBMS to Hadoop migration

Highlighted

RDBMS to Hadoop migration

Contributor

clarification 1:-

we used sqoop to migrate the data for all tables from RDBMS to hadoop.Then we performed query analysis using hive and finally we are generating reports using tableau.we are good till now.

Intially for RDBMS oracle database we used change capture stage in Datastage ETL to capture delta changes between two tables.I am not sure How to implement delta changes between two tables in hive since we migrated the Data from RDBMS to hadoopfor all tables?

clarification 2:-

can anybody suggest steps in RDBMS oracle tables to Hadoop migration?

6 REPLIES 6

Re: RDBMS to Hadoop migration

Super Guru
@vamsi valiveti

Sqoop has an incremental option. Please see the following link which describes doing CDC in Hadoop using Spark data frames.

https://community.hortonworks.com/articles/35709/basic-cdc-in-hadoop-using-spark-with-data-frames.ht...

Re: RDBMS to Hadoop migration

Contributor

@mqureshi

Thanks for input.only hive,pig,sqoop,oozie,hbase is allowed in my project as of now.Spark is not supported as of now

Re: RDBMS to Hadoop migration

Contributor

@vamsi valiveti, kindly find the below example to import using sqoop...

sqoop-import --connect jdbc:oracle://hostname:1521/schema --username <user_name> --password <pwd> --table <table_name> --hive-table schemaname.tablename --hive-import --hive-home /home/Hadoop/hive-0.11.0 -m 5

Re: RDBMS to Hadoop migration

Contributor

I am already using sqoop and i need inputs on below point

clarification 2:-
can anybody suggest steps in RDBMS oracle tables to Hadoop migration?

Re: RDBMS to Hadoop migration

@vamsi valiveti

If you are looking for import of all RDBMS Oracle tables onto Hadoop, then you can use sqoop import-all-tables.

Refer to link.

Re: RDBMS to Hadoop migration

Contributor

Thanks for input.It is a new project after importing the data into HDFS.what is the next step after migration.I am aware of that option of sqoop.currently i am in POC.we want to migrate the tables from RDBMS to hadoop.company do not want to use any RDBMS going further.

Don't have an account?
Coming from Hortonworks? Activate your account here