Sqoop Merge: Can't parse input data: '\N'

[#1]

Hi All,

I am getting error at Sqoop merge as below, Sqoop import is working fine for this table.

Can you please help? Any help is appreciated.

Error: java.lang.RuntimeException: Can't parse input data: '\N'
at table.__loadFromFields(table_class.java:1233)
at table.parse(table_class.java:1081)
at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:53)
at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:470)
at java.math.BigDecimal.<init>(BigDecimal.java:739)
at table.__loadFromFields(table_class.java:1190)
... 11 more

and Sqoop import command is as follows-

<start to="SQOOP_IMPORT_TABLE" />
<action name="SQOOP_IMPORT_TABLE">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<arg>import</arg>
<arg>-Dmapred.child.java.opts=-Xmx4096m</arg>
<arg>--connect</arg>
<arg>${database}</arg>
<arg>--query</arg>
<arg>${query}</arg>
<arg>--fields-terminated-by</arg>
<arg>'\001'</arg>
<arg>--username</arg>
<arg>${username}</arg>
<arg>--password-file</arg>
<arg>${password}</arg>
<arg>--null-string</arg>
<arg>\\n</arg>
<arg>--null-non-string</arg>
<arg>\\n</arg>
<arg>--target-dir</arg>
<arg>/user/${queueName}/landing/staging/${table}/${PERIOD_DATA_DATE}_uncompressed</arg>
<arg>-m</arg>
<arg>1</arg>
<arg>--map-column-java</arg>
<arg>WKT=String</arg>
<arg>--jar-file</arg>
<arg>${jar_file}</arg>
<arg>--class-name</arg>
<arg>TABLE</arg>
<file>../${conf_location}/${jar_file}#${jar_file}</file>
<file>../${conf_location}/${ojdbc_jar_file}#${ojdbc_jar_file}</file>
</sqoop>
<ok to="DELETE_UNCOMPRESSED_TABLE" />
<error to="Email_failure" />

Thanks,

[#2]

It looks like you have \n defined for null-non-string, but in the input it has a null as \N.

Changing the null-non-string to \\N should resolve the issue (unless you have both \n and \N in the source, in which case you need to cleanse your data).

Hopefully this helps!

[#3]

I have tried using \\N as well, its still giving same error. Input data from Oracle DB either have some value in it or it is null.

[#4]

My apologies, I misunderstood the question. You say the import is working for the table but sqoop-merge is failing. Are you running a merge against a dataset that had null defined as \N and one where null was defined as \n?

[#5]

running a merge against a dataset that had null defined as \N.