question Re: how to do incremental load using sqoop shell script? By passing parameters? in Archives of Support Questions (Read Only)

how to do incremental load using sqoop shell script? By passing parameters?

Former Member — Thu, 03 Nov 2016 20:10:32 GMT

+#!/bin/bash

echo " sqoop import --connect $connectors//$orginServer/$orginDatabase --table $origTable --username $username --password $password -m 1 --hive-import --hive-database $hiveDatabase --hive-table $myTable --target-dir /tmp/sqoop_1 --drivers $drivers"

My sample script is above how to do modification in above script for sqoop incremental using column time stamp and merge id

Re: how to do incremental load using sqoop shell script? By passing parameters?

bhagan — Sat, 05 Nov 2016 02:45:56 GMT

To do incremental imports, you need to add a few more arguments to your command. Take a look at the following description from the sqoop docs:

https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_incremental_imports

Inside your shell script, I’m not sure how you’re setting the values of your named parameters, unless you have more code that is mapping the positional parameters to local variables.

When using the lastmodified mode, it requires that a field in your table to be a timestamp, which will be updated with the current timestamp during update. Additionally, the update requires a check column, which can be of type timestamp also.

Re: how to do incremental load using sqoop shell script? By passing parameters?

ylu — Tue, 08 Nov 2016 23:58:25 GMT

It really depends on what your relational database is. Some relational database does not support incremental imports. Otherwise, the link above provided by bhagan should sufficient.