Multiple sqoop on same hive table

[#1 · Accepted solution]

@Gayathri Devi,

Assuming your initial data to be and Index as row key which will be increasing as new data is inserted

Index, value ---> col names
1,a
2,b
3,c

You can run normal sqoop command which will import the complete data to destination. Now let's say you have added few more rows to the source and your input becomes like this

Index, value ---> col names
1,a
2,b
3,c
4,d
5,e
6,f

Now you can use sqoop incremental command to import the new columns . You can use "--incremental<mode> --check-column<column name> --last-value<last check column value>"

ie "--incremental append --check-column Index --last-value 3". This command will only import the last 3 rows

You can also do incremental import based on lastmodified value.

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

Thanks,

Aditya