IllegalArgumentException and Illegal partition for 'val' in sqoop

simran_k — Sat, 02 Jul 2016 17:22:41 GMT

I have asked this question over and over and am starting to believe that I am missing out on something really basic here. Not a lot of people seem to have come across this and I am really stuck on this one:

I get this error when I specify merge key argument with incremental import lastmodified .

sqoop job --meta-connect jdbc:hsqldb:hsql://FQDN:16000/sqoop --create JOB_NAME -- import --driver com.mysql.jdbc.Driver --connect jdbc:mysql://IP/erp?zeroDateTimeBehavior=convertToNull --username USERNAME --password 'PASSWORD' --table TABLE_NAME --merge-key id --split-by id --incremental lastmodified --check-column update_date --last-value 0 --null-string '\\N' --null-non-string '\\N' --fields-terminated-by '\001' --hive-drop-import-delims

The first import works alright .Starting second import I get:

Error: java.io.IOException: Illegal partition for 1 (-2)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1083)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.sqoop.mapreduce.MergeMapperBase.processRecord(MergeMapperBase.java:82)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:58)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.lang.IllegalArgumentException
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:330)
	at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:51)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1848)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.io.IOException: Illegal partition for 67204 (-2)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1083)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.sqoop.mapreduce.MergeMapperBase.processRecord(MergeMapperBase.java:82)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:58)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Error: java.io.IOException: Illegal partition for 201610 (-2)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1083)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.sqoop.mapreduce.MergeMapperBase.processRecord(MergeMapperBase.java:82)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:58)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Why is it?What is really the solution?

In this error line, 1 is a value in the column that is specified in --merge-key arguement.

Illegal partition for 1

As suggested, I finally created a small test table to test with an int, datetime and varchar , without any NULL or invalid chars in the data and yet I faced the same issue:

sqoop job --meta-connect jdbc:hsqldb:hsql://FQDN:16000/sqoop  --create test_table -- import --driver com.mysql.jdbc.Driver --connect jdbc:mysql://IP/DB?zeroDateTimeBehavior=convertToNull --username USER_NAME --password 'PASSWORD' --table test_table --merge-key id --split-by id --target-dir LOCATION --incremental lastmodified --last-value 0 --check-column updated_at

There were only 2 rows in the data and yet I got this:

Error: java.lang.IllegalArgumentException
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:330)
	at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:51)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1848)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Error: java.io.IOException: Illegal partition for 3 (-2)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1083)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.sqoop.mapreduce.MergeMapperBase.processRecord(MergeMapperBase.java:82)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:58)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Error: java.lang.IllegalArgumentException
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:330)
	at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:51)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1848)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1508)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Error: java.io.IOException: Illegal partition for 1 (-2)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1083)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.sqoop.mapreduce.MergeMapperBase.processRecord(MergeMapperBase.java:82)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:58)
	at org.apache.sqoop.mapreduce.MergeTextMapper.map(MergeTextMapper.java:34)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Here's what my data looks like:

# id, updated_at, name 

'1', '2016-06-29 14:08:55', 'c' 

'3', '2016-06-29 14:12:53', 'f'

I missed out on a really important detail here that I get this error only in OOZIE and submit the job through HUE and this works just fine including the Merge mapreduce when I run the sqoop job through command line

Taken from oozie launcher, This is what my mapreduce job logs look like:

>>> Invoking Sqoop command line now >>>

5373 [uber-SubtaskRunner] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
5407 [uber-SubtaskRunner] INFO  org.apache.sqoop.Sqoop  - Running Sqoop version: 1.4.6-cdh5.7.0
5702 [uber-SubtaskRunner] WARN  org.apache.sqoop.tool.BaseSqoopTool  - Setting your password on the command-line is insecure. Consider using -P instead.
5715 [uber-SubtaskRunner] WARN  org.apache.sqoop.ConnFactory  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
5740 [uber-SubtaskRunner] WARN  org.apache.sqoop.ConnFactory  - Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
5754 [uber-SubtaskRunner] INFO  org.apache.sqoop.manager.SqlManager  - Using default fetchSize of 1000
5754 [uber-SubtaskRunner] INFO  org.apache.sqoop.tool.CodeGenTool  - Beginning code generation
6091 [uber-SubtaskRunner] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM test_table AS t WHERE 1=0
6098 [uber-SubtaskRunner] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM test_table AS t WHERE 1=0
6118 [uber-SubtaskRunner] INFO  org.apache.sqoop.orm.CompilationManager  - HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop-mapreduce
8173 [uber-SubtaskRunner] INFO  org.apache.sqoop.orm.CompilationManager  - Writing jar file: /tmp/sqoop-yarn/compile/454902ac78d49b783a1f51b7bfe0a2be/test_table.jar
8185 [uber-SubtaskRunner] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM test_table AS t WHERE 1=0
8192 [uber-SubtaskRunner] INFO  org.apache.sqoop.tool.ImportTool  - Incremental import based on column updated_at
8192 [uber-SubtaskRunner] INFO  org.apache.sqoop.tool.ImportTool  - Lower bound value: '2016-07-02 17:13:24.0'
8192 [uber-SubtaskRunner] INFO  org.apache.sqoop.tool.ImportTool  - Upper bound value: '2016-07-02 17:16:56.0'
8194 [uber-SubtaskRunner] INFO  org.apache.sqoop.mapreduce.ImportJobBase  - Beginning import of test_table
8214 [uber-SubtaskRunner] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM test_table AS t WHERE 1=0
8230 [uber-SubtaskRunner] WARN  org.apache.sqoop.mapreduce.JobBase  - SQOOP_HOME is unset. May not be able to find all job dependencies.
8716 [uber-SubtaskRunner] INFO  org.apache.sqoop.mapreduce.db.DBInputFormat  - Using read commited transaction isolation
8717 [uber-SubtaskRunner] INFO  org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - BoundingValsQuery: SELECT MIN(id), MAX(id) FROM test_table WHERE ( updated_at >= '2016-07-02 17:13:24.0' AND updated_at < '2016-07-02 17:16:56.0' )
8721 [uber-SubtaskRunner] INFO  org.apache.sqoop.mapreduce.db.IntegerSplitter  - Split size: 0; Num splits: 4 from: 1 to: 1
25461 [uber-SubtaskRunner] INFO  org.apache.sqoop.mapreduce.ImportJobBase  - Transferred 26 bytes in 17.2192 seconds (1.5099 bytes/sec)
25471 [uber-SubtaskRunner] INFO  org.apache.sqoop.mapreduce.ImportJobBase  - Retrieved 1 records.
25536 [uber-SubtaskRunner] WARN  org.apache.sqoop.mapreduce.ExportJobBase  - IOException checking input file header: java.io.EOFException
25550 [uber-SubtaskRunner] WARN  org.apache.sqoop.mapreduce.JobBase  - SQOOP_HOME is unset. May not be able to find all job dependencies.
Heart beat
Heart beat
70628 [uber-SubtaskRunner] ERROR org.apache.sqoop.tool.ImportTool  - Merge MapReduce job failed!
70628 [uber-SubtaskRunner] INFO  org.apache.sqoop.tool.ImportTool  - Saving incremental import state to the metastore
70831 [uber-SubtaskRunner] INFO  org.apache.sqoop.tool.ImportTool  - Updated data for job: test_table

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

pminovic — Sat, 02 Jul 2016 18:03:11 GMT

Try first to drop "hive-drop-import-delims" because it's a Hive specific option but you are not importing to Hive. Then make sure id is integer (to support split-by) and unique (to support merge). Check also my working example.

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

simran_k — Sat, 02 Jul 2016 18:11:30 GMT

@Predrag Minovic: I did try dropping hive-drop-import-delims , also id is a primary key and an auto incrementing integer, I am just out of ideas what could really be the problem

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

pminovic — Sat, 02 Jul 2016 18:23:28 GMT

Okay, try just import, without creating a job, and drop also "last-value", it's in a wrong format I think. If it still fails then drop "merge-key". Keep on dropping options until it works. Then try adding them one by one to localize the problem.

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

simran_k — Sat, 02 Jul 2016 18:41:42 GMT

@Predrag Minovic: Updated the question. Please have a look. Also, the problem arises only when I specify a merge key. Incremental append works alright but for incremental last modified, I do have to specify a merge key for the updated and that is exactly what isn't working .Also , if I run the sqoop job through command line, it works just fine. Also, please "note" that "I get this problem only while I run the job through oozie",

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

simran_k — Sat, 02 Jul 2016 19:14:25 GMT

Hey, just realized that it happens only when I submit job to oozie. It works fine when I run/execute sqoop jobs through the command line. I am submitting my jobs using hue. What could be the reason? @Predrag Minovic

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

simran_k — Sat, 02 Jul 2016 20:34:32 GMT

@Predrag Minovic: Let me know if you find anything. It looks like a bug to me

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

pminovic — Sun, 03 Jul 2016 04:10:43 GMT

Since we resolved your Sqoop commnad line issue, please close this question and start a new one about your Oozie Sqoop action, providing your workflow and other necessary details.

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

simran_k — Tue, 05 Jul 2016 12:14:07 GMT

Hey, Thanks but it always worked through CLI and never did from oozie. Should I still create a new issue for that? @Predrag Minovic

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

simran_k — Sat, 23 Jul 2016 18:35:28 GMT

Here's a workaround that fixed it for me: http://www.yourtechchick.com/sqoop/illegal-partition-exception-sqoop-incremental-imports/

question Re: IllegalArgumentException and Illegal partition for 'val' in sqoop in Archives of Support Questions (Read Only)

IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop

Re: IllegalArgumentException and Illegal partition for 'val' in sqoop