Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop: Incremental loads in Hive without a timestamp

Sqoop: Incremental loads in Hive without a timestamp

Explorer

Hello guys,

 

 Based on your help, I was able to create incremental loads in sqoop using Hive as a target database.

 

I use the lastmodified option of sqoop:

--incremental lastmodified

 

Unfortunately, the --incremental append mode doesn't work with Hive:

Append mode for imports is not compatible with HCatalog. Please remove the parameter--append-mode

 

We've found some source tables in SQL Server which don't have a datetime column, this situation prevents us to do incremental loads with sqoop.

 

Is there an alternative way to do incremental loads with those tables that don't have a datetime column? We cannot modify the source tables.

 

 

1 REPLY 1
Highlighted

Re: Sqoop: Incremental loads in Hive without a timestamp

Contributor

@Heri  Suggestion is to use triggers and create new table with the timestamp field and run the sqoop incremental job on the newly created table.

 

https://stackoverflow.com/questions/34806245/incremental-data-load-using-sqoop-without-primary-key-o...

Don't have an account?
Coming from Hortonworks? Activate your account here