My database has a timestamp column on the basis of which Iam performing my sqoop incremental import with last modified clause.
If I am giving last modified check column value as 11am it doesn't retrieves the records which were inserted at 11am it imports records after that.
How do I import the records processed at 11am.
I don't want to have any duplicate records or any missing records.
its is always recommended to run this as sqoop job so that you will have your last value being recorded automatically.
would you consider performing
which specifies the column to be examined when determining which rows to import.
will insert all the new rows based on the last value
I have table in sql server that column contain random unique number there is no any primary key but we want to perform incremental append or lastmodified operation using sqoop so please help me.
Note:-This is Critical Issue.
You can perform lastmodified option.
Something like the below
sqoop import \ --connect --username --password --table --incremental lastmodified \ --check cloumn last_updated_date_or anything that is according to your table --last--vaule " 2101-02-22 01:02:12"
I recall you don't need the column to be a date, but for squoop to know which records are added/changed after the point where you already got, you do need to have something incremental.
If you have no column that can be easily used to determine whether a row is newer or not, the only conceptual way to know whether a row is new, would be by keeping track of which values have already been loaded. This administration is very heavy something that tools like sqoop cannot do automatically.
Hi @DennisJaheruddi ,
Thanks for replied.
Problem is here to identify the newly added row inside sqoop as well as hdfs/hive.because whatever we are
getting that data from sqlserver there ie no any unique key value i.e each column contains duplicates data in a
random manner.i tried more to find out this solution but there is no luck from my sode.
Thanks and appreciated if you find some solution for this issue.