Support Questions
Find answers, ask questions, and share your expertise

Sqoop Import all tables along with updation into hive?

Hi all,

I have created 3 tables in one schema in mysql. I want to import all tables into hive directly.

initial load i imported mysql 3 tables into hive at 25/10/2016 10:30

i want next incremental load at 25/10/2016 12:00;

if any rows added to mysql tables after 25/10/2016 10:30 from that tables, i have to import newly added rows (delta load) into hive using sqoop.

please try solve this problem

thanks in advance

3 REPLIES 3

Re: Sqoop Import all tables along with updation into hive?

Guru

To import all tables use: sqoop import-all-tables

See:

https://www.tutorialspoint.com/sqoop/sqoop_import_all_tables.htm

To do added rows, for each table use: sqoop import with a --where clause for your timestamp column.

See:

https://www.tutorialspoint.com/sqoop/sqoop_import.htm

If you want to include records that have been modified since your initial load (not just added), then you need to use --check-column {modified_date_col} --incremental {lastmodified} --last-value {modified_date}

See section 7.2.7: https://sqoop.apache.org/docs/1.4.1-incubating/SqoopUserGuide.html

You will need to to merge the deltas with the intial dump for each table. See: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_dataintegration/content/incrementally-up...

Re: Sqoop Import all tables along with updation into hive?

hi

i need to automatically update tables directly from mysql table to hive table along with each and every time

i have to use same sqoop job for mutiple times in a day

please share me sample example

thanks in advance

swathi

Re: Sqoop Import all tables along with updation into hive?

only using sqoop import command