Reply
Explorer
Posts: 25
Registered: ‎04-26-2016

sqoop append passing conditional input values

I am using the below command to first time complete full load, it is automatically creating the table and loading the data.

 

Now if i want to just append the data to the table which has same structure as source, can you pleas provide the command to just append plus pasing input $conditions values how from back end linux CLI side.

 

sqoop import --connect jdbc:oracle:thin:@svrpdkp:1521:prod --username hadooppsvc --P --query "SELECT * FROM V500.med_admin_event WHERE \$CONDITIONS" --target-dir /user/hive/warehouse/research_stage.db/st_med_admin_event --hive-import --hive-database research_stage --hive-table st_med_admin_event --split-by MED_ADMIN_EVENT_ID

 

Thank you very much for the helpful info.

Champion
Posts: 564
Registered: ‎05-16-2016

Re: sqoop append passing conditional input values

[ Edited ]

sqoop comes with 
 append mode or lastmodified mode when it comes to incremental import.

 

it is a good idea to run this type of incremenatl import in job so that the last-value for a subsequent import will be stored and you dont have to mark down in the print screen.

 

 

 

example 

table student 
id   name  age  date  
100 xxx 12 10/2/2015 keep the id as --check-coloumn (contains row id ) keep the date as --lastmodified sqoop import \ --connect jdbc:mysql://localhost/userdb \ --username user1 \ --table student \ --m 1 \ --incremental append \ --check-column id \ -last value 100
Explorer
Posts: 25
Registered: ‎04-26-2016

Re: sqoop append passing conditional input values

Hi Guna,

 

Thanks for your response, i want to pass date value in WHERE \$CONDITIONS within query .

 

i want to sqoop all rows where update_dt > $CONDITIONS

 

with incremental append, is it possible.

 

Because i have more than 50 table objects, that way i can pass the WHERE \$CONDITIONS value dynamically thru a shell script. to run sequential sqoop incremental process.

 

Thank you very much for the helpful info.

Highlighted
Champion
Posts: 564
Registered: ‎05-16-2016

Re: sqoop append passing conditional input values

[ Edited ]

Well $CONDITION is basically used for internal process of sqoop to set boundary when it is runining in parallel . 
 

if understand correctly the below sqoop command shoud be suffice . correct me if I am wrong . 

 

Also I did not understand the table object that you are mentioning . 

 

 

sqoop import
 --connect jdbc:mysql://localhost:3306/meta
--table Junk --username
--password --check-column update_dt  --incremental lastmodified --last-value 2014-01-25
-target-dir

 

or 

 

you can use --where clause 

 

 

 !/bin/bash
value = ${^^1}

sqoop import \ --connect jdbc:mysql://localhost/userdb \--username root \ --table emp \ --m 1 \ --where updated_dt > ${value} \ --target-dir

 

Announcements