Bug in Sandbox 2.3.2 hitting SQOOP-1400

[#1]

Hitting https://issues.apache.org/jira/browse/SQOOP-1400 on Sandbox 2.3.2 when executing

sqoop export --connect jdbc:mysql://127.0.0.1/export --username hive --password hive --table exported --export-dir /apps/hive/warehouse/export_table

to overcome the issue add the following

--driver com.mysql.jdbc.Driver

mysql-connector-java-5.1.17 needs to be updated

[#2 · Accepted solution]

We should be using mysql-connector-java-5.1.29.jar or above. There is an internal JIRA BUG-38264 tracking this, please reopen that.

[#3]

I believe this is specific to Sandbox 2.3.2

[#4]

@Artem Ervits I have accepted Deepesh response as Best Answer. Please let me know your feedback

[#5]

I don't have access to internal JIRAs. As far as a workaround, when you do yum update, OS will report that v. 17 is the latest even though we ship v. 29 in hdp-utils repo. Removing the connector and replacing with v. 29 will not work either, I tried it. Adding the --driver flag is the only way I was able to get it to work. I do also agree we should ship later version. The whole reason for this post is to include SQOOP-1400 in the next release of Sandbox. What is more surprising is that we ship Sqoop 1.4.6 and that should include the fix but seems it doesn't? @Neeraj

@Deepesh

[#6]

Hi Artem,

In my sandbox version:

Created on: 27_10_2015_13_22_00 for virtualbox                                                                                                                                                                              
Hadoop stack version:  Hadoop 2.7.1.2.3.2.0-2950                                                                                                                                                                            
Ambari Version: 2.1.2                                                                                                                                                                                                       
Ambari Hash: 0ef0b7b62cf14eaaff3c5c3f416253f568f323f9                                                                                                                                                                       
Ambari build:  Release : 377                                                                                                                                                                                                

OS Version: CentOS release 6.7 (Final)

I extracted /usr/hdp/current/sqoop-server/lib//usr/hdp/current/sqoop-server/lib/mysql-connector-java.jar and the INDEX.LIST shows mysql-connector-java-5.1.31-bin.jar . So the version should be correct. @artem Did you make any changes to the sandbox or is unchanged from downloading?

Even with the above jar, still reproduce Artem's behavior, bug filed BUG-47916

[#7]

@rmolina@hortonworks.com absolutely unchanged, please run "rpm -qa | grep mysql-connector-java", RHEL version of the mysql connector always supersedes the version provided in HDP-UTILS, even if HDP-UTILS version is higher. Uninstalling the connector and installing the one from HDP-UTILS, which is v. 29 still didn't make a difference. There's something else at play besides the version issues. SQOOP-1400 is included in Sqoop 1.4.6 and we ship 1.4.6 so I can't explain the problem. Here's an easy way to recreate the issue on a vanilla Sandbox VM.

# Create Hive table

drop table if exists export_table;

create table export_table (

key int,

value string

)

row format delimited

fields terminated by ",";

# populate Hive with dummy data

insert into export_table values("1", "ExportedValue");

# confirm Hive table has data

select * from export_table;

# export table to MySQL

# MySQL table must exist

su mysql

mysql -u root

create database export;

use export;

create table exported (rowkey int, value varchar(20));

exit;

# on HDP 2.3.2 Sandbox, SQOOP-1400 bug, use --driver com.mysql.jdbc.Driver to overcome the problem

# sqoop export from a Hive table into MySQL

sqoop export --connect jdbc:mysql://127.0.0.1/export --username hive --password hive --table exported --direct --export-dir /apps/hive/warehouse/export_table --driver com.mysql.jdbc.Driver

if you exclude the last flag "--driver com.mysql.jdbc.Driver" error will occur.

[#8]

Hi @Artem Ervits appreciate it, I already reproduced and filed BUG-47916. Interesting in my tests of the sandbox I have HDP-UTILS repo and got 5.1.17 which also am discussing with teams. (removed all repos from /etc/yum.repos.d except HDP-UTILS and executed yum clean all then yum info mysql)

[#9]

I believe the issue is that RHEL comes bundled with mysql-connector-java or at least until you try to install mysql then it grabs the package from the RHEL repo ignoring the package from our repo. If there's a way to handle the logic so that our repo takes precedence, then I think we should be good.

[#10]

I've just encountered this issue when using the latest version of the HDP sandbox on the Oracle VM on a Mac. I specified the driver into the sqoop command line options and the data was escorted. However, the output from sqoop contained the following warning that as the driver options had been specified, then sqoop had fallen back to the generic connection manager and it suggested that I should specify a specific connection manager.

Is it worth worrying about this warning and if so, what connection manager corresponds to the driver ?