Created 09-13-2017 03:23 PM
Hi All,
I am getting error at Sqoop merge as below, Sqoop import is working fine for this table.
Can you please help? Any help is appreciated.
Error: java.lang.RuntimeException: Can't parse input data: '\N'
at table.__loadFromFields(table_class.java:1233)
at table.parse(table_class.java:1081)
at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:53)
at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:470)
at java.math.BigDecimal.<init>(BigDecimal.java:739)
at table.__loadFromFields(table_class.java:1190)
... 11 more
and Sqoop import command is as follows-
<start to="SQOOP_IMPORT_TABLE" />
<action name="SQOOP_IMPORT_TABLE">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<arg>import</arg>
<arg>-Dmapred.child.java.opts=-Xmx4096m</arg>
<arg>--connect</arg>
<arg>${database}</arg>
<arg>--query</arg>
<arg>${query}</arg>
<arg>--fields-terminated-by</arg>
<arg>'\001'</arg>
<arg>--username</arg>
<arg>${username}</arg>
<arg>--password-file</arg>
<arg>${password}</arg>
<arg>--null-string</arg>
<arg>\\n</arg>
<arg>--null-non-string</arg>
<arg>\\n</arg>
<arg>--target-dir</arg>
<arg>/user/${queueName}/landing/staging/${table}/${PERIOD_DATA_DATE}_uncompressed</arg>
<arg>-m</arg>
<arg>1</arg>
<arg>--map-column-java</arg>
<arg>WKT=String</arg>
<arg>--jar-file</arg>
<arg>${jar_file}</arg>
<arg>--class-name</arg>
<arg>TABLE</arg>
<file>../${conf_location}/${jar_file}#${jar_file}</file>
<file>../${conf_location}/${ojdbc_jar_file}#${ojdbc_jar_file}</file>
</sqoop>
<ok to="DELETE_UNCOMPRESSED_TABLE" />
<error to="Email_failure" />
Thanks,
Created 09-13-2017 05:49 PM
It looks like you have \n defined for null-non-string, but in the input it has a null as \N.
Changing the null-non-string to \\N should resolve the issue (unless you have both \n and \N in the source, in which case you need to cleanse your data).
Hopefully this helps!
Created 09-13-2017 06:15 PM
Created 09-13-2017 06:47 PM
My apologies, I misunderstood the question. You say the import is working for the table but sqoop-merge is failing. Are you running a merge against a dataset that had null defined as \N and one where null was defined as \n?
Created 10-10-2017 10:45 AM
running a merge against a dataset that had null defined as \N.