Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to run a sqoop command inside the java code

Highlighted

How to run a sqoop command inside the java code

New Contributor

I want to move data from Oracle to hive or HDFS. I have done import using sqoop command. But i want to do it through java code. Please let me know how to do this. I have tried few codes in internet but nothing worked. Please let me know the flow to do this.

 

5 REPLIES 5

Re: How to run a sqoop command inside the java code

Master Collaborator

@bas I don't have any example code for you, but would suggest that you browse the Java Client API Guide and possibly even pick up a copy of the Apache Sqoop Cookbook ?  

 

If you've got some example code you've tried to run and are getting some sort of error, can we see that so we can help troubleshoot?

 

Regards

Re: How to run a sqoop command inside the java code

New Contributor

Hi Clint,

 

I have successfully moved data from oracle to HDFS using the below code (stackoverflow code http://stackoverflow.com/questions/9229611/how-to-use-sqoop-in-java-program). Now the data is stored as part-m-00000 file in HDFS

1.I will download that file

2. create a table in hive manually.

3. Load this downloaded file into the hive table.

 

But i want to directly move data from java code using sqoop to the hive. I know there is --hive-import in sqoop command line,  but i am not gettig how to use this --hive-import in my jode code. Please let me know.  Here is the code

 

import com.cloudera.sqoop.SqoopOptions;

import com.cloudera.sqoop.tool.ImportTool;

public class SqoopJavaInterface {
    public static void main(String[] args) {
        SqoopOptions options = new SqoopOptions();
        options.setConnectString("jdbc:oracle:thin:@192.168.0.129:1521/xe");
        options.setTableName("test3");
        options.setColumns(new String[]{"firstname", "lastname"});
        //options.setWhereClause("id>10");     // this where clause works when importing whole table, ie when setTableName() is used
        options.setUsername("***");
        options.setPassword("*****");
        options.setSplitByCol("firstname");
        //options.setDirectMode(true);    // Make sure the direct mode is off when importing data to HBase
        options.setNumMappers(8);         // Default value is 4
        options.setSqlQuery("SELECT * FROM test3");
            int ret = new ImportTool().run(options);
    }
}
 They have used the code below to import for Hbase, but i want to import for Hive. How to do this.
 
// HBase options        options.setHBaseTable("HBASE_TABLE_NAME");        options.setHBaseColFamily("colFamily");        options.setCreateHBaseTable(true);// Create HBase table, if it does not exist        options.setHBaseRowKeyColumn("log_id");
 

Re: How to run a sqoop command inside the java code

Cloudera Employee

Sqoop 1.x do not have official Java API and direct use of SqoopOptions and ImportTool classes is not recommended. The problem with such use is that you need to ensure that all configuration, dependencies and environment is set up correctly prior calling the Sqoop classes. I would recommend to stick with the "sqoop" binary that is shipped with CDH as this binary will set up all required.

Re: How to run a sqoop command inside the java code

Explorer

Hello, does Oozie action "sqoop" is officially supported? And if it is, then how to run sqoop action with desired user? Because even if I run whole oozie workflow as specific user and provide --hive-import, table in hive is generated as hive user although files that were imported from PSQL are owned by correct user. That gives me an error, that files could not be moved from tmp location to hive warehouse, because owners in both locations does not match.

P.S. should I create new thread for this?

Re: How to run a sqoop command inside the java code

New Contributor

What are the jar files you used for this code? Can you plz share link for the same.