Created 08-28-2017 04:56 PM
My database has a timestamp column on the basis of which Iam performing my sqoop incremental import with last modified clause.
But
If I am giving last modified check column value as 11am it doesn't retrieves the records which were inserted at 11am it imports records after that.
How do I import the records processed at 11am.
I don't want to have any duplicate records or any missing records.
Created 08-28-2017 07:45 PM
its is always recommended to run this as sqoop job so that you will have your last value being recorded automatically.
would you consider performing
--incremental append
with
--check-column
which specifies the column to be examined when determining which rows to import.
will insert all the new rows based on the last value
Created 05-15-2019 11:55 PM
Dear All.
I have table in sql server that column contain random unique number there is no any primary key but we want to perform incremental append or lastmodified operation using sqoop so please help me.
Note:-This is Critical Issue.
Thanks
HadoopHelp
Created 05-20-2019 11:29 PM
You can perform lastmodified option.
Something like the below
sqoop import \ --connect --username --password --table --incremental lastmodified \ --check cloumn last_updated_date_or anything that is according to your table --last--vaule " 2101-02-22 01:02:12"
Created 05-20-2019 11:31 PM
Thanks!
but there is no any time/date column in my table?
then how can we perform last-modified operation?
Thanks
HadoopHelp
Created 06-04-2019 01:53 AM
I recall you don't need the column to be a date, but for squoop to know which records are added/changed after the point where you already got, you do need to have something incremental.
If you have no column that can be easily used to determine whether a row is newer or not, the only conceptual way to know whether a row is new, would be by keeping track of which values have already been loaded. This administration is very heavy something that tools like sqoop cannot do automatically.
Created 07-26-2019 08:17 AM
Hi @DennisJaheruddi ,
Thanks for replied.
Problem is here to identify the newly added row inside sqoop as well as hdfs/hive.because whatever we are
getting that data from sqlserver there ie no any unique key value i.e each column contains duplicates data in a
random manner.i tried more to find out this solution but there is no luck from my sode.
Thanks and appreciated if you find some solution for this issue.
Thanks
HadoopHelp
Created 07-29-2019 01:14 AM
Created 05-20-2019 11:29 PM
Let me know if you need any more information.