Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Multiple sqoop on same hive table

Solved Go to solution
Highlighted

Multiple sqoop on same hive table

New Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Multiple sqoop on same hive table

@Gayathri Devi,

Assuming your initial data to be and Index as row key which will be increasing as new data is inserted

Index, value ---> col names
1,a
2,b
3,c

You can run normal sqoop command which will import the complete data to destination. Now let's say you have added few more rows to the source and your input becomes like this

Index, value ---> col names
1,a
2,b
3,c
4,d
5,e
6,f

Now you can use sqoop incremental command to import the new columns . You can use "--incremental<mode> --check-column<column name> --last-value<last check column value>"

ie "--incremental append --check-column Index --last-value 3". This command will only import the last 3 rows

You can also do incremental import based on lastmodified value.

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

Thanks,

Aditya

1 REPLY 1

Re: Multiple sqoop on same hive table

@Gayathri Devi,

Assuming your initial data to be and Index as row key which will be increasing as new data is inserted

Index, value ---> col names
1,a
2,b
3,c

You can run normal sqoop command which will import the complete data to destination. Now let's say you have added few more rows to the source and your input becomes like this

Index, value ---> col names
1,a
2,b
3,c
4,d
5,e
6,f

Now you can use sqoop incremental command to import the new columns . You can use "--incremental<mode> --check-column<column name> --last-value<last check column value>"

ie "--incremental append --check-column Index --last-value 3". This command will only import the last 3 rows

You can also do incremental import based on lastmodified value.

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

Thanks,

Aditya

Don't have an account?
Coming from Hortonworks? Activate your account here