Member since
08-12-2016
2
Posts
0
Kudos Received
0
Solutions
03-13-2017
08:59 PM
Hi @Rahul Sounder I am using incremental append for updates as well as new inserts and its working fine for me. sqoop job -Dmapreduce.job.name=<job_name> --create <job_name> --meta-connect ${metaconnect_db} -- import --connect <db connection string> \--username <USER_NAME> \--password <PWD> \--query <Query WHERE \$CONDITIONS> \--as-textfile \--fields-terminated-by <DELIMITER_VALUE> \--target-dir <TGT_DR> \--m <MAPPER_CNT> \--null-string '\\N' \--null-non-string '\\N' \--check-column <check_column> \--incremental append \--hive-delims-replacement " " \--split-by <check_column> \--last-value <LAST VALUE>; , Hi @Rahul Sounder I am using incremental append mode and its working for me. sqoop job -Dmapreduce.job.name=<job_name> --create <job_name> --meta-connect ${metaconnect_db} -- import --connect <db connection string> \--username <USER_NAME> \--password <PWD> \--query <Query WHERE \$CONDITIONS> \--as-textfile \--fields-terminated-by <DELIMITER_VALUE> \--target-dir <TGT_DR> \--m <MAPPER_CNT> \--null-string '\\N' \--null-non-string '\\N' \--check-column <check_column> \--incremental append \--hive-delims-replacement " " \--split-by <check_column> \--last-value <LAST VALUE>;
... View more
08-12-2016
03:43 PM
@Predrag Minovic Hi, I was trying to do the incremental import using the incremental last modified and check column. last value stored in sqoop metastore was system timestamp instead of Max(Check column) value. Example: Check column - timestamp field - Max value for that field in table - 2016-08-08 21:08:19.813 Initial load went fine without any issues. After initial load last value got updated to "2016-08-12 10:40:22.627" instead of "2016-08-08 21:08:19.813" due to this I am missing records inserted after 2016-08-08 21:08:19.813. Thanks, Thiru , Hi Predrag Minovic, I was trying to do the Sqoop incremental import using the incremental last modified. last value get stored in sqoop metastore is system timestamp instead of Max(check_column). Thanks, Thiru
... View more