Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop Error when importing large data sets

Highlighted

Sqoop Error when importing large data sets

Hi

I am trying to ingest data from DB2 to hdfs using sqoop. I was able to successfully load data for with a small date range ie in months range . But when trying the import with large date range say for 16 years ,I am facing following exception

Caused by: com.ibm.db2.jcc.a.SqlException: [jcc][t4][1065][12306][4.3.111] Caught java.io.CharConversionException.  See attached Throwable for details. ERRORCODE=-4220, SQLSTATE=null
        at com.ibm.db2.jcc.a.dd.a(dd.java:660)
        at com.ibm.db2.jcc.a.dd.a(dd.java:60)
        at com.ibm.db2.jcc.a.dd.a(dd.java:112)
        at com.ibm.db2.jcc.a.ic.a(ic.java:1853)
        at com.ibm.db2.jcc.a.ic.n(ic.java:504)
        at com.ibm.db2.jcc.a.ic.F(ic.java:1285)
        at com.ibm.db2.jcc.a.yl.d(yl.java:1038)
        at com.ibm.db2.jcc.a.yl.getString(yl.java:1016)
        at org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
        at com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
        at QueryResult.readFields(QueryResult.java:8240)
        at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
        ... 12 more
Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:282)
        at com.ibm.db2.jcc.a.u.a(u.java:44)
        at com.ibm.db2.jcc.a.ic.a(ic.java:1845)
        ... 20 more

Is it because of encountering any non UTF-8 data or anything related to the data size?

Work arounds did so far

1.Edited mapred-site.xml and set "mapreduce.map.java.opts" to -Xmx1024m -Ddb2.jcc.charsetDecoderEncoder=3 -> Reference

2.Executed the same query in db2 with same date range - Worked fine

Any help is highly appreciated

Thanks in advance

Aparna

5 REPLIES 5
Highlighted

Re: Sqoop Error when importing large data sets

Expert Contributor

@aparna, this CharConversionExceptionexact error can be resolved by using exact jars please refer the below link. Hope this will be useful.

http://stackoverflow.com/questions/16910791/getting-error-code-4220-with-null-sql-state

Thanks!

Highlighted

Re: Sqoop Error when importing large data sets

Thanks for the reply

But then why didn't it failed for smaller data set ?

I am facing this issue only for large data imports

Thanks

Aparna

Highlighted

Re: Sqoop Error when importing large data sets

Expert Contributor

Hi Aparna,

After using exact jars also facing same issue with large data set? I'm not sure about how it's worked for small data set will check and if I found solution will let you know

Thanks!

Mahesh

Highlighted

Re: Sqoop Error when importing large data sets

New Contributor

Did your workaround 1 solve your problem, or was that an unsuccessful attempt? It's interesting because i'm having the same issue, yet I was able to sqoop the same date range a few days ago but it was split-by a different column. Thanks!

Re: Sqoop Error when importing large data sets

New Contributor

Using bullet (1.) we were able to fix the issue!

Don't have an account?
Coming from Hortonworks? Activate your account here