Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎10-19-2017

What is the best way to run a sqoop import from java?

I am building a java application that is suppoed to import data via sqoop and do some hdfs operations.

This application should run on Cloudera CHD5.12.0 and sqoop 1.4.6

 

I successfully ran the sqoop import using a java ProcessBuilder. But this sees like a dirty way to do this.

 

My next approach was to use the sqoop ImortTool. But that way i ran into depreciated issues... I used org.apache.sqoop.tool.ImportTool which accepts a SqoopOptions object in the constructor. But it needs the SqoopOptions from com.cloudera... Package, which is depreciated.

 

So I tried a differen Solution:

import org.apache.sqoop.Sqoop;
import org.apache.hadoop.conf.Configuration;

Configuration conf = new Configuration();
Sqoop.runTool(sqoopOptions.toArray(new String[sqoopOptions.size()]), conf);

This results in folloing Error:

17/10/18 11:01:15 WARN tool.SqoopTool: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
17/10/18 11:01:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.12.0
17/10/18 11:01:15 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/10/18 11:01:15 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
17/10/18 11:01:15 INFO manager.SqlManager: Using default fetchSize of 1000
17/10/18 11:01:15 ERROR oracle.OracleConnectionFactory: Unable to load the jdbc driver class : oracle.jdbc.OracleDriver
17/10/18 11:01:15 ERROR tool.BaseSqoopTool: Got error creating database manager: java.lang.RuntimeException: Unable to load the jdbc driver class : oracle.jdbc.OracleDriver
	at org.apache.sqoop.manager.oracle.OracleConnectionFactory.loadJdbcDriver(OracleConnectionFactory.java:75)
	at org.apache.sqoop.manager.oracle.OracleConnectionFactory.createOracleJdbcConnection(OracleConnectionFactory.java:52)
	at org.apache.sqoop.manager.oracle.OraOopConnManager.makeConnection(OraOopConnManager.java:97)
	at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
	at org.apache.sqoop.manager.oracle.OraOopManagerFactory.accept(OraOopManagerFactory.java:114)
	at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:184)
	at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:270)
	at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:95)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:609)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
	at jobs.util.FullTableImportStrategy.runSqoopImportToHdfsTempDir(FullTableImportStrategy.java:77)
	at jobs.util.FullTableImportStrategy.execute(FullTableImportStrategy.java:29)
	at jobs.ImportJob.<init>(ImportJob.java:12)
	at start.Main.main(Main.java:12)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Oracle JDBC Driver is set up correctly, since everything works from commandline and with the ProcessBuilder.

 

How can i resolve this error, and is there maybe a better way to run a sqoop command from a java program?

Thank you for your help!

Cloudera Employee
Posts: 211
Registered: ‎03-23-2015

Re: What is the best way to run a sqoop import from java?

The Oracle JDBC driver JAR file is probably not setup in the java class path, have you checked that?
Announcements