Multiple sqoop on same hive table

Gayathridevi — Fri, 10 Nov 2017 16:09:53 GMT

Re: Multiple sqoop on same hive table

asirna — Fri, 10 Nov 2017 16:33:41 GMT

Assuming your initial data to be and Index as row key which will be increasing as new data is inserted

Index, value ---> col names
1,a
2,b
3,c

You can run normal sqoop command which will import the complete data to destination. Now let's say you have added few more rows to the source and your input becomes like this

Index, value ---> col names
1,a
2,b
3,c
4,d
5,e
6,f

Now you can use sqoop incremental command to import the new columns . You can use "--incremental<mode> --check-column<column name> --last-value<last check column value>"

ie "--incremental append --check-column Index --last-value 3". This command will only import the last 3 rows

You can also do incremental import based on lastmodified value.

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

Thanks,

Aditya

question Multiple sqoop on same hive table in Archives of Support Questions (Read Only)

Multiple sqoop on same hive table

Re: Multiple sqoop on same hive table