Support Questions
Find answers, ask questions, and share your expertise

Incremental import in HDFS

Contributor

I need to import data from mysql, at the end of everyday to my hdfs. I have scheduled my sqoop action using oozie.

my sqoop command will put data in a new directory every time. I need to name this directory as it won't be repeated after the first time. so I can not provide a constant --target-dir. I tried to use 'current-date'. so every day it creates a new directory but this function is not supported in xml 1.0 and oozie does not support version 2.0.

do you know any other way to create an incremental and unique id?

2 REPLIES 2

Contributor

@Sara Alizadeh

You can use Falcon to run sqoop import periodically .Please find below link which explains this in detail.

https://falcon.apache.org/ImportExport.html

Did you ever come up with a workable solution for this? If so, would you mind sharing what you settled on? Thanks!