Reply
rio
Explorer
Posts: 48
Registered: ‎04-18-2014

Sqoop with duplicate column name

Hello,

 

Not sure if this is a known bug but I am unable to import data into Hive from MySQL using sqoop.

 

I wrote a sqoop with duplicate column name (have alias) but it threw me an error message "Duplicate Column identifier specified: 'id'".

I modified sqoop to have concat function and now it gives me an error "Hive does not support the SQL type for column a"

 

 

sqoop import \
--connect jdbc:mysql://foo.test.net/mfg \
--username pingp \
--password 987yjd \
--hive-import \
--hive-table third_map \
--query "select concat(r.id,'') a, concat(p.id,'') b from tblDimMfg r join tblDimMfg p on r.id = p.id where r.Name = 'bbp' and p.Name = 'bbt' and \$CONDITIONS" \
--target-dir /user/test/hivehome/mysql/third_map \
--fields-terminated-by '\t' \
--hive-drop-import-delims \
-m 1

 

Could you please provide me suggestion?

 

Thank you,

Rio

Cloudera Employee
Posts: 35
Registered: ‎08-18-2014

Re: Sqoop with duplicate column name

How about this command:

 

sqoop import \
--connect jdbc:mysql://foo.test.net/mfg \
--username pingp \
--password 987yjd \
--hive-import \
--hive-table third_map \
--query "select r.id as a, p.id as b from tblDimMfg r join tblDimMfg p on r.id = p.id where r.Name = 'bbp' and p.Name = 'bbt' and \$CONDITIONS" \
--target-dir /user/test/hivehome/mysql/third_map \
--fields-terminated-by '\t' \
--hive-drop-import-delims \
-m 1

 

Announcements
New solutions