Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Incremental import in HDFS

Incremental import in HDFS

Contributor

I need to import data from mysql, at the end of everyday to my hdfs. I have scheduled my sqoop action using oozie.

my sqoop command will put data in a new directory every time. I need to name this directory as it won't be repeated after the first time. so I can not provide a constant --target-dir. I tried to use 'current-date'. so every day it creates a new directory but this function is not supported in xml 1.0 and oozie does not support version 2.0.

do you know any other way to create an incremental and unique id?

2 REPLIES 2
Highlighted

Re: Incremental import in HDFS

Contributor

@Sara Alizadeh

You can use Falcon to run sqoop import periodically .Please find below link which explains this in detail.

https://falcon.apache.org/ImportExport.html

Re: Incremental import in HDFS

Did you ever come up with a workable solution for this? If so, would you mind sharing what you settled on? Thanks!