Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

sqoop append passing conditional input values

sqoop append passing conditional input values

Explorer

I am using the below command to first time complete full load, it is automatically creating the table and loading the data.

 

Now if i want to just append the data to the table which has same structure as source, can you pleas provide the command to just append plus pasing input $conditions values how from back end linux CLI side.

 

sqoop import --connect jdbc:oracle:thin:@svrpdkp:1521:prod --username hadooppsvc --P --query "SELECT * FROM V500.med_admin_event WHERE \$CONDITIONS" --target-dir /user/hive/warehouse/research_stage.db/st_med_admin_event --hive-import --hive-database research_stage --hive-table st_med_admin_event --split-by MED_ADMIN_EVENT_ID

 

Thank you very much for the helpful info.

3 REPLIES 3

Re: sqoop append passing conditional input values

Champion

sqoop comes with 
 append mode or lastmodified mode when it comes to incremental import.

 

it is a good idea to run this type of incremenatl import in job so that the last-value for a subsequent import will be stored and you dont have to mark down in the print screen.

 

 

 

example 

table student 
id   name  age  date  
100 xxx 12 10/2/2015 keep the id as --check-coloumn (contains row id ) keep the date as --lastmodified sqoop import \ --connect jdbc:mysql://localhost/userdb \ --username user1 \ --table student \ --m 1 \ --incremental append \ --check-column id \ -last value 100
Highlighted

Re: sqoop append passing conditional input values

Explorer

Hi Guna,

 

Thanks for your response, i want to pass date value in WHERE \$CONDITIONS within query .

 

i want to sqoop all rows where update_dt > $CONDITIONS

 

with incremental append, is it possible.

 

Because i have more than 50 table objects, that way i can pass the WHERE \$CONDITIONS value dynamically thru a shell script. to run sequential sqoop incremental process.

 

Thank you very much for the helpful info.

Re: sqoop append passing conditional input values

Champion

Well $CONDITION is basically used for internal process of sqoop to set boundary when it is runining in parallel . 
 

if understand correctly the below sqoop command shoud be suffice . correct me if I am wrong . 

 

Also I did not understand the table object that you are mentioning . 

 

 

sqoop import
 --connect jdbc:mysql://localhost:3306/meta
--table Junk --username
--password --check-column update_dt  --incremental lastmodified --last-value 2014-01-25
-target-dir

 

or 

 

you can use --where clause 

 

 

 !/bin/bash
value = ${^^1}

sqoop import \ --connect jdbc:mysql://localhost/userdb \--username root \ --table emp \ --m 1 \ --where updated_dt > ${value} \ --target-dir