question Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400 in Archives of Support Questions (Read Only)

Bug in Sandbox 2.3.2 hitting SQOOP-1400

aervits — Fri, 06 Nov 2015 00:34:07 GMT

Hitting https://issues.apache.org/jira/browse/SQOOP-1400 on Sandbox 2.3.2 when executing

sqoop export --connect jdbc:mysql://127.0.0.1/export --username hive --password hive --table exported --export-dir /apps/hive/warehouse/export_table

to overcome the issue add the following

--driver com.mysql.jdbc.Driver

mysql-connector-java-5.1.17 needs to be updated

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

deepesh1 — Fri, 06 Nov 2015 00:49:51 GMT

We should be using mysql-connector-java-5.1.29.jar or above. There is an internal JIRA BUG-38264 tracking this, please reopen that.

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

aervits — Fri, 06 Nov 2015 01:17:26 GMT

I believe this is specific to Sandbox 2.3.2

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

nsabharwal — Fri, 06 Nov 2015 20:01:49 GMT

@Artem Ervits I have accepted Deepesh response as Best Answer. Please let me know your feedback

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

aervits — Sun, 08 Nov 2015 06:01:22 GMT

I don't have access to internal JIRAs. As far as a workaround, when you do yum update, OS will report that v. 17 is the latest even though we ship v. 29 in hdp-utils repo. Removing the connector and replacing with v. 29 will not work either, I tried it. Adding the --driver flag is the only way I was able to get it to work. I do also agree we should ship later version. The whole reason for this post is to include SQOOP-1400 in the next release of Sandbox. What is more surprising is that we ship Sqoop 1.4.6 and that should include the fix but seems it doesn't? @Neeraj

@Deepesh

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

rmolina — Thu, 12 Nov 2015 07:27:04 GMT

Hi Artem,

In my sandbox version:

Created on: 27_10_2015_13_22_00 for virtualbox                                                                                                                                                                              
Hadoop stack version:  Hadoop 2.7.1.2.3.2.0-2950                                                                                                                                                                            
Ambari Version: 2.1.2                                                                                                                                                                                                       
Ambari Hash: 0ef0b7b62cf14eaaff3c5c3f416253f568f323f9                                                                                                                                                                       
Ambari build:  Release : 377

OS Version: CentOS release 6.7 (Final)

I extracted /usr/hdp/current/sqoop-server/lib//usr/hdp/current/sqoop-server/lib/mysql-connector-java.jar and the INDEX.LIST shows mysql-connector-java-5.1.31-bin.jar . So the version should be correct. @artem Did you make any changes to the sandbox or is unchanged from downloading?

Even with the above jar, still reproduce Artem's behavior, bug filed BUG-47916

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

aervits — Thu, 12 Nov 2015 09:50:13 GMT

@rmolina@hortonworks.com absolutely unchanged, please run "rpm -qa | grep mysql-connector-java", RHEL version of the mysql connector always supersedes the version provided in HDP-UTILS, even if HDP-UTILS version is higher. Uninstalling the connector and installing the one from HDP-UTILS, which is v. 29 still didn't make a difference. There's something else at play besides the version issues. SQOOP-1400 is included in Sqoop 1.4.6 and we ship 1.4.6 so I can't explain the problem. Here's an easy way to recreate the issue on a vanilla Sandbox VM.

# Create Hive table

drop table if exists export_table;

create table export_table (

key int,

value string

)

row format delimited

fields terminated by ",";

# populate Hive with dummy data

insert into export_table values("1", "ExportedValue");

# confirm Hive table has data

select * from export_table;

# export table to MySQL

# MySQL table must exist

su mysql

mysql -u root

create database export;

use export;

create table exported (rowkey int, value varchar(20));

exit;

# on HDP 2.3.2 Sandbox, SQOOP-1400 bug, use --driver com.mysql.jdbc.Driver to overcome the problem

# sqoop export from a Hive table into MySQL

sqoop export --connect jdbc:mysql://127.0.0.1/export --username hive --password hive --table exported --direct --export-dir /apps/hive/warehouse/export_table --driver com.mysql.jdbc.Driver

if you exclude the last flag "--driver com.mysql.jdbc.Driver" error will occur.

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

rmolina — Thu, 12 Nov 2015 11:31:52 GMT

Hi @Artem Ervits appreciate it, I already reproduced and filed BUG-47916. Interesting in my tests of the sandbox I have HDP-UTILS repo and got 5.1.17 which also am discussing with teams. (removed all repos from /etc/yum.repos.d except HDP-UTILS and executed yum clean all then yum info mysql)

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

aervits — Thu, 12 Nov 2015 22:10:34 GMT

I believe the issue is that RHEL comes bundled with mysql-connector-java or at least until you try to install mysql then it grabs the package from the RHEL repo ignoring the package from our repo. If there's a way to handle the logic so that our repo takes precedence, then I think we should be good.

Re: Bug in Sandbox 2.3.2 hitting SQOOP-1400

Jerry_Cllns — Fri, 12 Feb 2016 23:30:23 GMT

I've just encountered this issue when using the latest version of the HDP sandbox on the Oracle VM on a Mac. I specified the driver into the sqoop command line options and the data was escorted. However, the output from sqoop contained the following warning that as the driver options had been specified, then sqoop had fallen back to the generic connection manager and it suggested that I should specify a specific connection manager.

Is it worth worrying about this warning and if so, what connection manager corresponds to the driver ?