I am trying to ingest data from DB2 to hdfs using sqoop. I was able to successfully load data for with a small date range ie in months range . But when trying the import with large date range say for 16 years ,I am facing following exception
Caused by: com.ibm.db2.jcc.a.SqlException: [jcc][t4][4.3.111] Caught java.io.CharConversionException. See attached Throwable for details. ERRORCODE=-4220, SQLSTATE=null at com.ibm.db2.jcc.a.dd.a(dd.java:660) at com.ibm.db2.jcc.a.dd.a(dd.java:60) at com.ibm.db2.jcc.a.dd.a(dd.java:112) at com.ibm.db2.jcc.a.ic.a(ic.java:1853) at com.ibm.db2.jcc.a.ic.n(ic.java:504) at com.ibm.db2.jcc.a.ic.F(ic.java:1285) at com.ibm.db2.jcc.a.yl.d(yl.java:1038) at com.ibm.db2.jcc.a.yl.getString(yl.java:1016) at org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71) at com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61) at QueryResult.readFields(QueryResult.java:8240) at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244) ... 12 more Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1 at java.nio.charset.CoderResult.throwException(CoderResult.java:282) at com.ibm.db2.jcc.a.u.a(u.java:44) at com.ibm.db2.jcc.a.ic.a(ic.java:1845) ... 20 more
Is it because of encountering any non UTF-8 data or anything related to the data size?
Work arounds did so far
1.Edited mapred-site.xml and set "mapreduce.map.java.opts" to -Xmx1024m -Ddb2.jcc.charsetDecoderEncoder=3 -> Reference
2.Executed the same query in db2 with same date range - Worked fine
Any help is highly appreciated
Thanks in advance
@aparna, this CharConversionExceptionexact error can be resolved by using exact jars please refer the below link. Hope this will be useful.
Thanks for the reply
But then why didn't it failed for smaller data set ?
I am facing this issue only for large data imports
After using exact jars also facing same issue with large data set? I'm not sure about how it's worked for small data set will check and if I found solution will let you know
Did your workaround 1 solve your problem, or was that an unsuccessful attempt? It's interesting because i'm having the same issue, yet I was able to sqoop the same date range a few days ago but it was split-by a different column. Thanks!