Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Sqoop mysql connection manager

avatar
Expert Contributor

When running below sqoop import from mysql database

 

sqoop import \
--driver com.mysql.jdbc.Driver \
--connect jdbc:mysql://localhost/loudacre \
--username cloudera --password cloudera \
--table accounts \
--split-by acct_num \
--target-dir sqoop/accounts

 

I got the following warning (it seems Sqoop can't find a specific connection manager for MySQL driver, thus it uses the GenericJdbcManager):

 

WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.

Thus I've tried to specify the --connection-manager like so:

 

sqoop import \
--driver com.mysql.jdbc.Driver \
--connection-manager org.apache.sqoop.manager.MySQLManager \ --connect jdbc:mysql://localhost/loudacre \ --username cloudera --password cloudera \ --table accounts \ --split-by acct_num \ --target-dir sqoop/accounts

 

but I got error

 

ERROR sqoop.ConnFactory: Sqoop wasn't able to create connnection manager properly. Some of the connectors supports explicit --driver and some do not. Please try to either specify --driver or leave it out.
18/01/12 03:10:43 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: java.lang.NoSuchMethodException: org.apache.sqoop.manager.MySQLManager.<init>(java.lang.String, com.cloudera.sqoop.SqoopOptions)
	at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:171)
	at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:258)
	at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:89)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:593)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.NoSuchMethodException: org.apache.sqoop.manager.MySQLManager.<init>(java.lang.String, com.cloudera.sqoop.SqoopOptions)
	at java.lang.Class.getConstructor0(Class.java:3082)
	at java.lang.Class.getDeclaredConstructor(Class.java:2178)
	at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:157)

It seems Sqoop is not able to find org.apache.sqoop.manager.MySQLManager class. What do I need to do to make it available to Sqoop? Or am I forced to use the GenericJdbcManager? 

 

Thanks for any info

6 REPLIES 6

avatar
Champion

Do you have a mysql jdbc jar in the lib directory of the sqoop ? 

were you able to succefully perform sqoop import / export command ? 

avatar
Expert Contributor

Yes I have the mysql connector jar in the  /usr/lib/sqoop/lib/ folder. The import without the --connection-manager parameter finish successfully, but with the warning that the GenericJdbcManager is used by Sqoop, so I was wondering if I need to specify the --connection-manager explicitly to get the best performance for a mysql import

avatar
Champion

 

@cjervis  - This is the third time i'm posting the below message in the same thread as it get deleted automatically everytime when I edit the below message. In summary, I made few typos and tried to correct it by editing the message but found the entire post is missing after edit. Please take care!!

 

@ludof

 

You have mentioned that you are importing data from MySQL and you have already installed MySQL Connector. So sqoop suppose to use MySQL Connector but it is still using "Generic JDBC Connector". I believe the below three reasons will be the reason for your issue

 

1. You might have made some mistake in the MySQL Connector configuration/installation


(or)

 

2. The MySQL Connector version that you are using is not Suitable for your MySQL Version.
Ex: If you are using MySQL 5.6, it requires a driver version 5.1.26 or higher


(or)

 

3. May be you would get the latest mysql connector version in /usr/share/java, still it may create trouble if you have old version and new version in the same folder and mysql-connector-java.jar is still pointing to the older version

Ex: Consider you have below old & new version in /usr/share/java and mysql-connector-java.jar is pointing to older version
mysql-connector-java-5.1.25-bin.jar -- old version
mysql-connector-java-5.1.40-bin.jar -- new version

then remove the link mysql-connector-java.jar and make it refer to the new version as follows
rm mysql-connector-java.jar
ln -s /usr/share/java/mysql-connector-java-5.1.40-bin.jar mysql-connector-java.jar
mysql-connector-java.jar -> /usr/share/java/mysql-connector-java-5.1.40-bin.jar

 

avatar
Community Manager

Sorry about that @saranvisa. It seems that the community's spam filter grabbed your previous replies to this thread for some reason. If I happens again, I'll dig deeper into it. 


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Expert Contributor

Hi, I've replaced all my mysql-connector*.jar files in the system to the last version: mysql-connector-java-5.1.45-bin.jar, but I got always the same warning.

 

Also I found that if I not specify the driver

--driver com.mysql.jdbc.Driver \

I got an error:

18/01/15 01:32:26 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/01/15 01:32:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/01/15 01:32:26 INFO teradata.TeradataManagerFactory: Loaded connector factory for 'Cloudera Connector Powered by Teradata' on version 1.6c5
18/01/15 01:32:26 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: No manager for connect string: jdbc:mysql://localhost/loudacre
	at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:191)
	at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:258)
	at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:89)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:593)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

Thus I was wondering if I need to specify other settings to proper configure the mysql connector...According to here https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cdh_ig_jdbc_driver_install.html I just need to put the .jar connector in the lib folder of Sqoop, but maybe there is something more to do?

 

Thanks for the help 

avatar
New Contributor

Do not specify driver in sqoop arguments.

Using the –driver parameter will always force Sqoop to use the Generic JDBC Connector regardless of if a more specialized connector is available. For example, if the MySQL specialized connector would be used because the URL starts with jdbc:mysql://, specifying the –driver option will force Sqoop to use the generic connector instead. As a result, in most cases, you should not need to use the –driver option at all.

 

 

Thanks,
Ankit Gaurav Sinha