Reply
New Contributor
Posts: 4
Registered: ‎11-04-2014

Handling bad records during sqoop import or export

I looked at the options provided by sqoop export operation but could not find any options to handle bad records. For example once in a while it is possible that a character is present where a number is expected in a huge set of records. Is there a way to handle these scenarios in sqoop without failing the job and providing the bad records in a file.

 

Even if this is out of the box functionality is there any work around by extending the code? 

 

Thanks,

 

 

Cloudera Employee abe
Cloudera Employee
Posts: 109
Registered: ‎08-08-2013

Re: Handling bad records during sqoop import or export

You can just type cast your numbers to strings. This will vary from database to database. You'll need to use a "free form" query. See http://sqoop.apache.org/docs/1.4.5/SqoopUserGuide.html#_free_form_query_imports for more info.
Highlighted
New Contributor
Posts: 4
Registered: ‎11-04-2014

Re: Handling bad records during sqoop import or export

number to string was just an example, I am just saying bad records in general. 

 

Another example would be length of value is too long. Do i have to handle each of those individually? Currently it rolls back all the import/export  when it hits a bad record

Announcements
New solutions