Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop Incremental Import Over Avro file

Sqoop Incremental Import Over Avro file

Contributor

Hello Experts,

Really looking for your help as we are stuck with one of the activities.

We have a HDP-2.5.3 hadoop cluster and we are ingesting data from source Oracle database using Sqoop through a shell script. We are importing data from Oracle as AVRO file and then creating Hive external tables pointing to those Avro. Now our use case is to migrate the incremental changes from source Oracle to Hadoop using the lastmodified parameter through using timestamp. But we are stuck here as can't find any suitable example on incremental Sqoop import over Avro files. Need your help to give an example on how to do this.

Really looking for help as we are now stuck.

Thanks and Regards,

Rajdip

4 REPLIES 4

Re: Sqoop Incremental Import Over Avro file

Contributor

Also one update, we are not using NiFi. This is a shell based utility we are using.

Re: Sqoop Incremental Import Over Avro file

Contributor

Hello Guys, really looking for some guidance here on how to handle the scenario?

Highlighted

Re: Sqoop Incremental Import Over Avro file

Contributor

Hi Guys,

Can you please guide for this issue? Really looking for some guidance here.

Re: Sqoop Incremental Import Over Avro file

@rajdip chaudhuri

For now Sqoop does not support incremental on Avro files, this is tracked from Jira SQOOP-1094.

One of the approach is to have a cronjob as:

1. Import sqoop incremental to a Hive table.

2. Create and then insert into avro table from the hive table from previous step.

Don't have an account?
Coming from Hortonworks? Activate your account here