Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop incremental import issue:parameter --check-column

Sqoop incremental import issue:parameter --check-column

New Contributor

mysql> desc departments -> ;

+---------+--------------------+

Field | Type | Null | Key | Default | Extra |

+---------+--------------------+

| dept_no | int(4) | YES | | NULL | |

| dept_name | varchar(40) | YES | | NULL |

mysql> select * from departments;

+---------+--------------------+

| dept_no | dept_name |

+---------+--------------------+

| d009 | Customer Service |

| d005 | Development |

| d002 | Finance |

| d003 | Human Resources |

| d001 | Marketing |

| d004 | Production |

| d006 | Quality Management |

| d008 | Research |

| d007 | Sales |

| d010 | test department |

+---------+--------------------+

Database hivedb has been created in the hive.

Full import run successfully.

hive> desc departments;

OK

dept_no string

dept_name string

Now I need to execute the incremental import..

sqoop import --connect jdbc:mysql://XXX:3306/employees --username root --password XXXX --table departments --hive-import --hive-table hivedb.departments --incremental append --check-column dept_no --target-dir /user/root/sqoop/dept;

......

......

18/11/19 11:39:46 INFO tool.ImportTool: Maximal id query for free form incremental import: SELECT MAX(`dept_no`) FROM `departments` 18/11/19 11:39:46 ERROR tool.ImportTool: Import failed: Character column (dept_no) can not be used to determine which rows to incrementally import.

How this error can be avoid, does the --check-column parameter has the limitation? How the incremental import can be done?

2 REPLIES 2
Highlighted

Re: Sqoop incremental import issue:parameter --check-column

@Prashant Ahire,

Yes. The incremental import is not supported for CHAR,VARCHAR data types. You can use integer or date datatypes for -check-column.

Highlighted

Re: Sqoop incremental import issue:parameter --check-column

@Prashant Ahire,

If this helped , please take a moment to login and "Accept" the answer. This will be useful for other community users.

Don't have an account?
Coming from Hortonworks? Activate your account here