Support Questions

Find answers, ask questions, and share your expertise

Monitor Historical files and transfer to destination based on date


I have data flow which is checking folder /users/data/${now():format("yyyyMMdd")} and it would pick all files under this folder and upload to S3 /users/aws/${now():format("yyyyMMdd")}. Suppose i get file today which is 3 days old in the folder /users/data/20200331 i need to scan that folder and pick that file as well and move to S3 user /users/aws/20200331. 

Note : File names doesnt have date, Files are just dropped inside date folders.







@Gubbi I think you would want to just add another check for now minus 1,2,3 days?  This seems most easy.   However I would wonder if you would just want to get everything from /users/data? Then the initial check doesn't care about now or today,  it just tries to get anything new in that folder.




@stevenmatison : We dont have control on /users/data folder. Vendor would publish files in folder datewise. So there is no easy way to scan historical files without using minus expression. Minus would restrict for certain days, What if i have file a week back and i need to pick that one too.. There is no definite date when the old files would be pushed to that folder. 

@Gubbi   You need to figure out how to use ListFile to get everything, then check the dates/filenames/folders after the files arrived.  Regardless of what folders the files are in.   This way if a folder from 3 days ago appears tomorrow, you will still download the files.


Can you scan /user/data/ and take all folders and files?



Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.