question Re: why do I get error during sqoop import every time i will not set the mapper to 1, in Archives of Support Questions (Read Only)

why do I get error during sqoop import every time i will not set the mapper to 1,

dcedielunesa — Fri, 16 Sep 2022 13:01:02 GMT

I'm importing 5.6 GB table

and the error:

Error: java.io.IOException: mysqldump terminated with status 2 at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:485) at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

RahulSoni — Fri, 23 Mar 2018 10:36:39 GMT

@Christian Lunesa

Can you please share your sqoop command? Are you using --direct by any chance?

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

dcedielunesa — Fri, 23 Mar 2018 11:08:35 GMT

import -Dorg.apache.sqoop.splitter.allow_text_splitter=true --connect jdbc:mysql://x.x.x.x:xxxx/kpcustomers --username root --password ******* --table customers --fields-terminated-by | -m 5 --hive-import --hive-overwrite --hive-table testing.customers --direct --verbose

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

RahulSoni — Fri, 23 Mar 2018 11:22:17 GMT

Are you using "split-by" column while not setting number of mappers to 1?

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

dcedielunesa — Fri, 23 Mar 2018 11:26:41 GMT

yes I am using --direct

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

RahulSoni — Fri, 23 Mar 2018 11:34:08 GMT

Also, can you please share the "actual" MR job logs that you can see when you are running your job with multiple mappers?

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

dcedielunesa — Fri, 23 Mar 2018 11:46:47 GMT

NO, I don't use split-by when I set mappers to 1 or in any

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

dcedielunesa — Fri, 23 Mar 2018 11:54:04 GMT

here's the stderr and stdout

stdout.txt & stderr.txt

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

dcedielunesa — Fri, 23 Mar 2018 11:57:03 GMT

The import will succeed if I will set the mappers to 1. I also notice that when I'm not using 1 mapper the yarn memory will be fully consumed

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

RahulSoni — Fri, 23 Mar 2018 13:18:28 GMT

So the issue is with the "PK" column used in distributing the data in case of multiple mappers. It has always been recommended that an "integral" column is used as the "split by" column and your import is trying to use the column "CustID" which is String.

Have a look at how your splits are calculated during the import.

8020 [main] WARN  org.apache.sqoop.mapreduce.db.TextSplitter  - You are strongly encouraged to choose an integral split column.
8025 [main] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - Creating input split with lower bound '`CustID` >= '1'' and upper bound '`CustID` < '3?????''
8025 [main] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - Creating input split with lower bound '`CustID` >= '1'' and upper bound '`CustID` < '3?????''
8025 [main] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - Creating input split with lower bound '`CustID` >= '3?????'' and upper bound '`CustID` < '5?????''
8025 [main] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - Creating input split with lower bound '`CustID` >= '3?????'' and upper bound '`CustID` < '5?????''
8025 [main] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - Creating input split with lower bound '`CustID` >= '5?????'' and upper bound '`CustID` < '7*?????''
8025 [main] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - Creating input split with lower bound '`CustID` >= '5?????'' and upper bound '`CustID` < '7*?????''
8025 [main] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - Creating input split with lower bound '`CustID` >= '7*?????'' and upper bound '`CustID` <= '999999''
8025 [main] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat  - Creating input split with lower bound '`CustID` >= '7*?????'' and upper bound '`CustID` <= '999999''
8068 [main] INFO  org.apache.hadoop.mapreduce.JobSubmitter  - number of splits:4

The "?" indicates some foreign characters probably not parsed properly and hence resulted in the failure of your tasks.

However, when you have only a single mapper, there is no such parse needed for CustID column and the data is "copied and pasted" to HDFS and your job ends OK.

Re: why do I get error during sqoop import every time i will not set the mapper to 1,

dcedielunesa — Sat, 24 Mar 2018 05:00:34 GMT

Okey, Thanks a lot for explaining it to me very well. . I really have a lot of things needed to learn for this job.