Support Questions
Find answers, ask questions, and share your expertise

how to do incremental load using sqoop shell script? By passing parameters?

+#!/bin/bash
echo " sqoop import --connect $connectors//$orginServer/$orginDatabase --table $origTable --username $username --password $password -m 1 --hive-import --hive-database $hiveDatabase --hive-table $myTable --target-dir /tmp/sqoop_1 --drivers $drivers"

My sample script is above how to do modification in above script for sqoop incremental using column time stamp and merge id

1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to do incremental load using sqoop shell script? By passing parameters?

Expert Contributor

To do incremental imports, you need to add a few more arguments to your command. Take a look at the following description from the sqoop docs:

https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_incremental_imports

Inside your shell script, I’m not sure how you’re setting the values of your named parameters, unless you have more code that is mapping the positional parameters to local variables.

When using the lastmodified mode, it requires that a field in your table to be a timestamp, which will be updated with the current timestamp during update. Additionally, the update requires a check column, which can be of type timestamp also.

View solution in original post

2 REPLIES 2

Re: how to do incremental load using sqoop shell script? By passing parameters?

Expert Contributor

To do incremental imports, you need to add a few more arguments to your command. Take a look at the following description from the sqoop docs:

https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_incremental_imports

Inside your shell script, I’m not sure how you’re setting the values of your named parameters, unless you have more code that is mapping the positional parameters to local variables.

When using the lastmodified mode, it requires that a field in your table to be a timestamp, which will be updated with the current timestamp during update. Additionally, the update requires a check column, which can be of type timestamp also.

View solution in original post

Re: how to do incremental load using sqoop shell script? By passing parameters?

Expert Contributor

It really depends on what your relational database is. Some relational database does not support incremental imports. Otherwise, the link above provided by bhagan should sufficient.