Reply
bas
New Contributor
Posts: 6
Registered: ‎10-18-2013

How to run a sqoop command inside the java code

I want to move data from Oracle to hive or HDFS. I have done import using sqoop command. But i want to do it through java code. Please let me know how to do this. I have tried few codes in internet but nothing worked. Please let me know the flow to do this.

 

Posts: 416
Topics: 51
Kudos: 75
Solutions: 49
Registered: ‎06-26-2013

Re: How to run a sqoop command inside the java code

@bas I don't have any example code for you, but would suggest that you browse the Java Client API Guide and possibly even pick up a copy of the Apache Sqoop Cookbook ?  

 

If you've got some example code you've tried to run and are getting some sort of error, can we see that so we can help troubleshoot?

 

Regards

bas
New Contributor
Posts: 6
Registered: ‎10-18-2013

Re: How to run a sqoop command inside the java code

[ Edited ]

Hi Clint,

 

I have successfully moved data from oracle to HDFS using the below code (stackoverflow code http://stackoverflow.com/questions/9229611/how-to-use-sqoop-in-java-program). Now the data is stored as part-m-00000 file in HDFS

1.I will download that file

2. create a table in hive manually.

3. Load this downloaded file into the hive table.

 

But i want to directly move data from java code using sqoop to the hive. I know there is --hive-import in sqoop command line,  but i am not gettig how to use this --hive-import in my jode code. Please let me know.  Here is the code

 

import com.cloudera.sqoop.SqoopOptions;

import com.cloudera.sqoop.tool.ImportTool;

public class SqoopJavaInterface {
    public static void main(String[] args) {
        SqoopOptions options = new SqoopOptions();
        options.setConnectString("jdbc:oracle:thin:@192.168.0.129:1521/xe");
        options.setTableName("test3");
        options.setColumns(new String[]{"firstname", "lastname"});
        //options.setWhereClause("id>10");     // this where clause works when importing whole table, ie when setTableName() is used
        options.setUsername("***");
        options.setPassword("*****");
        options.setSplitByCol("firstname");
        //options.setDirectMode(true);    // Make sure the direct mode is off when importing data to HBase
        options.setNumMappers(8);         // Default value is 4
        options.setSqlQuery("SELECT * FROM test3");
            int ret = new ImportTool().run(options);
    }
}
 They have used the code below to import for Hbase, but i want to import for Hive. How to do this.
 
// HBase options        options.setHBaseTable("HBASE_TABLE_NAME");        options.setHBaseColFamily("colFamily");        options.setCreateHBaseTable(true);// Create HBase table, if it does not exist        options.setHBaseRowKeyColumn("log_id");
 
Cloudera Employee
Posts: 20
Registered: ‎07-08-2013

Re: How to run a sqoop command inside the java code

Sqoop 1.x do not have official Java API and direct use of SqoopOptions and ImportTool classes is not recommended. The problem with such use is that you need to ensure that all configuration, dependencies and environment is set up correctly prior calling the Sqoop classes. I would recommend to stick with the "sqoop" binary that is shipped with CDH as this binary will set up all required.

Explorer
Posts: 12
Registered: ‎11-21-2013

Re: How to run a sqoop command inside the java code

[ Edited ]

Hello, does Oozie action "sqoop" is officially supported? And if it is, then how to run sqoop action with desired user? Because even if I run whole oozie workflow as specific user and provide --hive-import, table in hive is generated as hive user although files that were imported from PSQL are owned by correct user. That gives me an error, that files could not be moved from tmp location to hive warehouse, because owners in both locations does not match.

P.S. should I create new thread for this?

Highlighted
New Contributor
Posts: 1
Registered: ‎06-28-2017

Re: How to run a sqoop command inside the java code

What are the jar files you used for this code? Can you plz share link for the same.

Announcements