Reply
Contributor
Posts: 54
Registered: ‎11-24-2017

Sqoop mysql connection manager

When running below sqoop import from mysql database

 

sqoop import \
--driver com.mysql.jdbc.Driver \
--connect jdbc:mysql://localhost/loudacre \
--username cloudera --password cloudera \
--table accounts \
--split-by acct_num \
--target-dir sqoop/accounts

 

I got the following warning (it seems Sqoop can't find a specific connection manager for MySQL driver, thus it uses the GenericJdbcManager):

 

WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.

Thus I've tried to specify the --connection-manager like so:

 

sqoop import \
--driver com.mysql.jdbc.Driver \
--connection-manager org.apache.sqoop.manager.MySQLManager \ --connect jdbc:mysql://localhost/loudacre \ --username cloudera --password cloudera \ --table accounts \ --split-by acct_num \ --target-dir sqoop/accounts

 

but I got error

 

ERROR sqoop.ConnFactory: Sqoop wasn't able to create connnection manager properly. Some of the connectors supports explicit --driver and some do not. Please try to either specify --driver or leave it out.
18/01/12 03:10:43 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: java.lang.NoSuchMethodException: org.apache.sqoop.manager.MySQLManager.<init>(java.lang.String, com.cloudera.sqoop.SqoopOptions)
	at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:171)
	at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:258)
	at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:89)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:593)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.NoSuchMethodException: org.apache.sqoop.manager.MySQLManager.<init>(java.lang.String, com.cloudera.sqoop.SqoopOptions)
	at java.lang.Class.getConstructor0(Class.java:3082)
	at java.lang.Class.getDeclaredConstructor(Class.java:2178)
	at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:157)

It seems Sqoop is not able to find org.apache.sqoop.manager.MySQLManager class. What do I need to do to make it available to Sqoop? Or am I forced to use the GenericJdbcManager? 

 

Thanks for any info

Highlighted
Champion
Posts: 680
Registered: ‎05-16-2016

Re: Sqoop mysql connection manager

Do you have a mysql jdbc jar in the lib directory of the sqoop ? 

were you able to succefully perform sqoop import / export command ? 

Contributor
Posts: 54
Registered: ‎11-24-2017

Re: Sqoop mysql connection manager

Yes I have the mysql connector jar in the  /usr/lib/sqoop/lib/ folder. The import without the --connection-manager parameter finish successfully, but with the warning that the GenericJdbcManager is used by Sqoop, so I was wondering if I need to specify the --connection-manager explicitly to get the best performance for a mysql import

Posts: 440
Topics: 13
Kudos: 73
Solutions: 39
Registered: ‎09-02-2016

Re: Sqoop mysql connection manager

 

@cjervis  - This is the third time i'm posting the below message in the same thread as it get deleted automatically everytime when I edit the below message. In summary, I made few typos and tried to correct it by editing the message but found the entire post is missing after edit. Please take care!!

 

@ludof

 

You have mentioned that you are importing data from MySQL and you have already installed MySQL Connector. So sqoop suppose to use MySQL Connector but it is still using "Generic JDBC Connector". I believe the below three reasons will be the reason for your issue

 

1. You might have made some mistake in the MySQL Connector configuration/installation


(or)

 

2. The MySQL Connector version that you are using is not Suitable for your MySQL Version.
Ex: If you are using MySQL 5.6, it requires a driver version 5.1.26 or higher


(or)

 

3. May be you would get the latest mysql connector version in /usr/share/java, still it may create trouble if you have old version and new version in the same folder and mysql-connector-java.jar is still pointing to the older version

Ex: Consider you have below old & new version in /usr/share/java and mysql-connector-java.jar is pointing to older version
mysql-connector-java-5.1.25-bin.jar -- old version
mysql-connector-java-5.1.40-bin.jar -- new version

then remove the link mysql-connector-java.jar and make it refer to the new version as follows
rm mysql-connector-java.jar
ln -s /usr/share/java/mysql-connector-java-5.1.40-bin.jar mysql-connector-java.jar
mysql-connector-java.jar -> /usr/share/java/mysql-connector-java-5.1.40-bin.jar

 

Posts: 729
Kudos: 88
Solutions: 45
Registered: ‎04-06-2015

Re: Sqoop mysql connection manager

Sorry about that @saranvisa. It seems that the community's spam filter grabbed your previous replies to this thread for some reason. If I happens again, I'll dig deeper into it. 

Cy Jervis, Community Manager


Was your question answered? Make sure to mark it as an accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Contributor
Posts: 54
Registered: ‎11-24-2017

Re: Sqoop mysql connection manager

Hi, I've replaced all my mysql-connector*.jar files in the system to the last version: mysql-connector-java-5.1.45-bin.jar, but I got always the same warning.

 

Also I found that if I not specify the driver

--driver com.mysql.jdbc.Driver \

I got an error:

18/01/15 01:32:26 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/01/15 01:32:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/01/15 01:32:26 INFO teradata.TeradataManagerFactory: Loaded connector factory for 'Cloudera Connector Powered by Teradata' on version 1.6c5
18/01/15 01:32:26 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: No manager for connect string: jdbc:mysql://localhost/loudacre
	at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:191)
	at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:258)
	at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:89)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:593)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Thus I was wondering if I need to specify other settings to proper configure the mysql connector...According to here https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cdh_ig_jdbc_driver_install.html I just need to put the .jar connector in the lib folder of Sqoop, but maybe there is something more to do?

 

Thanks for the help 

Announcements
New solutions