Support Questions
Find answers, ask questions, and share your expertise

Sqoop Error when importing large data sets

Hi

I am trying to ingest data from DB2 to hdfs using sqoop. I was able to successfully load data for with a small date range ie in months range . But when trying the import with large date range say for 16 years ,I am facing following exception

Caused by: com.ibm.db2.jcc.a.SqlException: [jcc][t4][1065][12306][4.3.111] Caught java.io.CharConversionException.  See attached Throwable for details. ERRORCODE=-4220, SQLSTATE=null
        at com.ibm.db2.jcc.a.dd.a(dd.java:660)
        at com.ibm.db2.jcc.a.dd.a(dd.java:60)
        at com.ibm.db2.jcc.a.dd.a(dd.java:112)
        at com.ibm.db2.jcc.a.ic.a(ic.java:1853)
        at com.ibm.db2.jcc.a.ic.n(ic.java:504)
        at com.ibm.db2.jcc.a.ic.F(ic.java:1285)
        at com.ibm.db2.jcc.a.yl.d(yl.java:1038)
        at com.ibm.db2.jcc.a.yl.getString(yl.java:1016)
        at org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
        at com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
        at QueryResult.readFields(QueryResult.java:8240)
        at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
        ... 12 more
Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:282)
        at com.ibm.db2.jcc.a.u.a(u.java:44)
        at com.ibm.db2.jcc.a.ic.a(ic.java:1845)
        ... 20 more

Is it because of encountering any non UTF-8 data or anything related to the data size?

Work arounds did so far

1.Edited mapred-site.xml and set "mapreduce.map.java.opts" to -Xmx1024m -Ddb2.jcc.charsetDecoderEncoder=3 -> Reference

2.Executed the same query in db2 with same date range - Worked fine

Any help is highly appreciated

Thanks in advance

Aparna

5 REPLIES 5

Expert Contributor

@aparna, this CharConversionExceptionexact error can be resolved by using exact jars please refer the below link. Hope this will be useful.

http://stackoverflow.com/questions/16910791/getting-error-code-4220-with-null-sql-state

Thanks!

Thanks for the reply

But then why didn't it failed for smaller data set ?

I am facing this issue only for large data imports

Thanks

Aparna

Expert Contributor

Hi Aparna,

After using exact jars also facing same issue with large data set? I'm not sure about how it's worked for small data set will check and if I found solution will let you know

Thanks!

Mahesh

New Contributor

Did your workaround 1 solve your problem, or was that an unsuccessful attempt? It's interesting because i'm having the same issue, yet I was able to sqoop the same date range a few days ago but it was split-by a different column. Thanks!

New Contributor

Using bullet (1.) we were able to fix the issue!